2024-09-17T15:43:46.283 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T15:43:46.296 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T15:43:46.371 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909006 branch: wip-bharath10-testing-2024-09-17-1348 description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} email: skanta@redhat.com first_in_suite: false job_id: '7909006' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath10-testing-2024-09-17-1348 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 workunit: branch: wip-bharath10-testing-2024-09-17-1348 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 3421 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath10-testing-2024-09-17-1348 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 targets: smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPs6D+avoID64XaIDlhZNbFahlVlGaO/2yXNNopgu6Z0bYxO6caXnMFFvtAN8ZkiuIElrKpDsxf4tgpMiluP9FI= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - erasure-code teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_15:01:33 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T15:43:46.371 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa; will attempt to use it 2024-09-17T15:43:46.372 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks 2024-09-17T15:43:46.373 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T15:43:46.374 INFO:teuthology.task.internal:Checking packages... 2024-09-17T15:43:46.395 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1889a165b5128787ec693450a3d1c5df5d6dc203' 2024-09-17T15:43:46.396 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T15:43:46.396 INFO:teuthology.packaging:ref: None 2024-09-17T15:43:46.396 INFO:teuthology.packaging:tag: None 2024-09-17T15:43:46.396 INFO:teuthology.packaging:branch: wip-bharath10-testing-2024-09-17-1348 2024-09-17T15:43:46.396 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:43:46.396 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath10-testing-2024-09-17-1348 2024-09-17T15:43:46.536 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4975.g1889a165 2024-09-17T15:43:46.538 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T15:43:46.544 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T15:43:46.544 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T15:43:46.551 INFO:teuthology.task.internal:Saving configuration 2024-09-17T15:43:46.566 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T15:43:46.573 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T15:43:46.598 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909006', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 15:38:38.245495', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPs6D+avoID64XaIDlhZNbFahlVlGaO/2yXNNopgu6Z0bYxO6caXnMFFvtAN8ZkiuIElrKpDsxf4tgpMiluP9FI='} 2024-09-17T15:43:46.599 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T15:43:46.605 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-17T15:43:46.605 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T15:43:46.635 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1eba0f3d00>, signals=[15]) 2024-09-17T15:43:46.635 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T15:43:46.642 INFO:teuthology.task.internal:Opening connections... 2024-09-17T15:43:46.642 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-09-17T15:43:46.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:43:46.730 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T15:43:46.737 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-09-17T15:43:46.815 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-09-17T15:43:46.816 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-09-17T15:43:46.873 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2024-09-17T15:43:46.874 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="9" 2024-09-17T15:43:46.874 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2024-09-17T15:43:46.874 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2024-09-17T15:43:46.874 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="9" 2024-09-17T15:43:46.874 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el9" 2024-09-17T15:43:46.874 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T15:43:46.874 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-09-17T15:43:46.874 INFO:teuthology.orchestra.run.smithi162.stdout:LOGO="fedora-logo-icon" 2024-09-17T15:43:46.874 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T15:43:46.875 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2024-09-17T15:43:46.875 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T15:43:46.875 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T15:43:46.875 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T15:43:46.875 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-09-17T15:43:46.902 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T15:43:46.913 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T15:43:46.919 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T15:43:46.919 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-09-17T15:43:46.937 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T15:43:46.943 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T15:43:46.944 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-09-17T15:43:47.007 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T15:43:47.039 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T15:43:47.055 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T15:43:47.055 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T15:43:47.055 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T15:43:47.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:43:47.074 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-09-17T15:43:47.133 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-503.el9.x86_64 2024-09-17T15:43:47.133 INFO:teuthology.task.kernel:Running kernel on smithi162: 5.14.0-503.el9.x86_64 2024-09-17T15:43:47.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2024-09-17T15:43:50.878 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.3 MB 00:01 2024-09-17T15:43:52.613 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-17T15:43:56.888 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-17T15:43:58.339 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 80 kB/s | 19 kB 00:00 2024-09-17T15:43:59.085 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 36 MB/s | 23 MB 00:00 2024-09-17T15:44:04.540 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-17T15:44:06.463 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T15:44:06.463 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T15:44:06.551 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:44:06.562 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:44:06.562 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-09-17T15:44:06.562 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:44:06.562 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:44:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T15:44:06.563 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-17T15:44:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T15:44:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T15:44:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T15:44:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:44:06.563 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:44:06.563 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:44:06.563 INFO:teuthology.orchestra.run.smithi162.stdout:Install 4 Packages 2024-09-17T15:44:06.563 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:44:06.564 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 79 M 2024-09-17T15:44:06.564 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 126 M 2024-09-17T15:44:06.564 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:44:06.683 INFO:teuthology.orchestra.run.smithi162.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 517 kB/s | 36 kB 00:00 2024-09-17T15:44:07.116 INFO:teuthology.orchestra.run.smithi162.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-09-17T15:44:07.800 INFO:teuthology.orchestra.run.smithi162.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 29 MB/s | 35 MB 00:01 2024-09-17T15:44:08.125 INFO:teuthology.orchestra.run.smithi162.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 20 MB/s | 29 MB 00:01 2024-09-17T15:44:08.125 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:08.125 INFO:teuthology.orchestra.run.smithi162.stdout:Total 50 MB/s | 79 MB 00:01 2024-09-17T15:44:08.807 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:44:09.180 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:44:09.180 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:44:10.143 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:44:10.144 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:44:12.593 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:44:13.237 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:44:13.325 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:44:16.097 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:44:16.234 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:44:21.061 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:44:21.100 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:44:25.961 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:45:04.430 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:45:04.470 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:45:05.715 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:45:05.716 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:45:05.716 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:45:05.716 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T15:45:05.716 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T15:45:05.716 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T15:45:05.716 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T15:45:05.716 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:45:05.716 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:45:06.080 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:45:06.187 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:45:06.188 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:45:06.188 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T15:45:06.188 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T15:45:06.188 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T15:45:06.188 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T15:45:06.189 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi162.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T15:45:06.189 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:45:06.247 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:45:06.247 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:45:06.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo rpm -qi grub2-tools 2024-09-17T15:45:06.282 INFO:teuthology.orchestra.run.smithi162.stdout:Name : grub2-tools 2024-09-17T15:45:06.283 INFO:teuthology.orchestra.run.smithi162.stdout:Epoch : 1 2024-09-17T15:45:06.283 INFO:teuthology.orchestra.run.smithi162.stdout:Version : 2.06 2024-09-17T15:45:06.283 INFO:teuthology.orchestra.run.smithi162.stdout:Release : 82.el9 2024-09-17T15:45:06.283 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: x86_64 2024-09-17T15:45:06.283 INFO:teuthology.orchestra.run.smithi162.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T15:45:06.283 INFO:teuthology.orchestra.run.smithi162.stdout:Group : Unspecified 2024-09-17T15:45:06.283 INFO:teuthology.orchestra.run.smithi162.stdout:Size : 8274670 2024-09-17T15:45:06.283 INFO:teuthology.orchestra.run.smithi162.stdout:License : GPLv3+ 2024-09-17T15:45:06.283 INFO:teuthology.orchestra.run.smithi162.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T15:45:06.283 INFO:teuthology.orchestra.run.smithi162.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T15:45:06.283 INFO:teuthology.orchestra.run.smithi162.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T15:45:06.283 INFO:teuthology.orchestra.run.smithi162.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T15:45:06.284 INFO:teuthology.orchestra.run.smithi162.stdout:Packager : builder@centos.org 2024-09-17T15:45:06.284 INFO:teuthology.orchestra.run.smithi162.stdout:Vendor : CentOS 2024-09-17T15:45:06.284 INFO:teuthology.orchestra.run.smithi162.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T15:45:06.284 INFO:teuthology.orchestra.run.smithi162.stdout:Summary : Support tools for GRUB. 2024-09-17T15:45:06.284 INFO:teuthology.orchestra.run.smithi162.stdout:Description : 2024-09-17T15:45:06.284 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:45:06.284 INFO:teuthology.orchestra.run.smithi162.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T15:45:06.284 INFO:teuthology.orchestra.run.smithi162.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T15:45:06.284 INFO:teuthology.orchestra.run.smithi162.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T15:45:06.284 INFO:teuthology.orchestra.run.smithi162.stdout:hardware devices. 2024-09-17T15:45:06.284 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:45:06.284 INFO:teuthology.orchestra.run.smithi162.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T15:45:06.285 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T15:45:06.285 INFO:teuthology.task.kernel:Updating grub on smithi162 to boot 5.14.0-508.el9.x86_64 2024-09-17T15:45:06.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T15:45:06.923 INFO:teuthology.orchestra.run.smithi162.stderr:Generating grub configuration file ... 2024-09-17T15:45:08.254 INFO:teuthology.orchestra.run.smithi162.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T15:45:08.283 INFO:teuthology.orchestra.run.smithi162.stderr:done 2024-09-17T15:45:08.285 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-09-17T15:45:08.300 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.oVCqbCokY3 2024-09-17T15:45:08.301 DEBUG:teuthology.orchestra.run.smithi162:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.oVCqbCokY3 2024-09-17T15:45:08.368 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.oVCqbCokY3 2024-09-17T15:45:08.510 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.oVCqbCokY3 is 6KB 2024-09-17T15:45:08.559 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.oVCqbCokY3 2024-09-17T15:45:08.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T15:45:08.637 INFO:teuthology.orchestra.run.smithi162.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T15:45:08.637 INFO:teuthology.orchestra.run.smithi162.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T15:45:08.637 INFO:teuthology.orchestra.run.smithi162.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T15:45:08.637 INFO:teuthology.orchestra.run.smithi162.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T15:45:08.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T15:45:08.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo shutdown -r now 2024-09-17T15:45:38.841 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T15:45:38.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-17T15:45:38.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:46:13.437 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-09-17T15:46:22.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-17T15:46:22.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:46:40.833 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-09-17T15:46:52.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-17T15:46:52.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:46:53.199 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-09-17T15:46:53.629 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-17T15:46:53.630 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T15:46:53.630 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-09-17T15:46:53.647 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-508.el9.x86_64 2024-09-17T15:46:53.647 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T15:46:53.647 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T15:46:53.648 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T15:46:54.649 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T15:46:54.649 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T15:46:54.764 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-09-17T15:46:54.784 DEBUG:teuthology.parallel:result is None 2024-09-17T15:46:54.784 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T15:46:54.792 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T15:46:54.792 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T15:46:54.807 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T15:46:54.814 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T15:46:54.822 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T15:46:54.822 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T15:46:54.870 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T15:46:54.877 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T15:46:54.877 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T15:46:54.938 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:46:54.948 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:46:54.950 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T15:46:54.956 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T15:46:54.956 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T15:46:55.015 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T15:46:55.024 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T15:46:55.024 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T15:46:55.070 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:46:55.153 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:46:55.195 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:46:55.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:46:55.318 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-17T15:46:55.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T15:46:55.387 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-09-17T15:46:55.490 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T15:46:55.626 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T15:46:55.633 INFO:teuthology.task.internal:Starting timer... 2024-09-17T15:46:55.633 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T15:46:55.645 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T15:46:55.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-09-17T15:46:55.687 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-09-17T15:46:55.689 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T15:46:55.689 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-09-17T15:46:55.750 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-09-17T15:46:55.750 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi162.front.sepia.ceph.com': 'permissive'} 2024-09-17T15:46:55.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T15:46:55.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:46:55.816 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T15:46:55.816 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-09-17T15:46:55.889 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T15:46:55.900 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-17T15:46:55.903 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-17T15:46:56.032 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T15:46:56.043 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T15:46:56.044 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi162.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T15:50:43.702 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi162.front.sepia.ceph.com')] 2024-09-17T15:50:43.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-17T15:50:43.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:50:43.780 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-09-17T15:50:43.816 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-17T15:50:43.817 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T15:50:43.829 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T15:50:43.829 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T15:50:43.830 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:50:43.885 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T15:50:43.899 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T15:50:43.923 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-09-17T15:50:43.935 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-09-17T15:50:43.949 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T15:50:43.963 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T15:50:44.019 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-09-17T15:50:44.024 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T15:50:44.024 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-09-17T15:50:44.024 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:50:44.024 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:50:44.024 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:50:44.024 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:50:44.024 INFO:teuthology.run_tasks:Running task install... 2024-09-17T15:50:44.033 DEBUG:teuthology.task.install:project ceph 2024-09-17T15:50:44.033 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}} 2024-09-17T15:50:44.033 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'} 2024-09-17T15:50:44.033 INFO:teuthology.task.install:Using flavor: default 2024-09-17T15:50:44.039 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T15:50:44.039 INFO:teuthology.task.install:extra packages: [] 2024-09-17T15:50:44.039 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'tag': None, 'wait_for_package': False} 2024-09-17T15:50:44.039 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:50:44.181 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/ 2024-09-17T15:50:44.181 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4975.g1889a165 2024-09-17T15:50:44.292 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T15:50:44.292 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-17T15:50:44.292 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T15:50:44.318 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T15:50:44.318 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1889a165b5128787ec693450a3d1c5df5d6dc203/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T15:50:44.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T15:50:44.462 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T15:50:44.526 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2024-09-17T15:50:44.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2024-09-17T15:50:44.754 INFO:teuthology.orchestra.run.smithi162.stdout:45 files removed 2024-09-17T15:50:44.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw 2024-09-17T15:50:45.341 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 308 kB/s | 84 kB 00:00 2024-09-17T15:50:45.625 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 100 kB/s | 24 kB 00:00 2024-09-17T15:50:45.901 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-17T15:50:47.684 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.3 MB 00:01 2024-09-17T15:50:50.277 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 13 MB/s | 20 MB 00:01 2024-09-17T15:50:54.753 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.5 MB 00:00 2024-09-17T15:50:56.437 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 120 kB/s | 19 kB 00:00 2024-09-17T15:50:57.516 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-17T15:51:02.913 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-17T15:51:05.164 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:51:05.186 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:51:05.186 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-17T15:51:05.186 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:51:05.186 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:51:05.186 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 11 M 2024-09-17T15:51:05.186 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2024-09-17T15:51:05.186 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.5 M 2024-09-17T15:51:05.186 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.3 M 2024-09-17T15:51:05.186 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-17T15:51:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T15:51:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T15:51:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 22 M 2024-09-17T15:51:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:19.3.0-4975.g1889a165.el9 ceph 25 k 2024-09-17T15:51:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T15:51:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T15:51:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T15:51:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 756 k 2024-09-17T15:51:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-17T15:51:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T15:51:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T15:51:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T15:51:05.187 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 508 k 2024-09-17T15:51:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T15:51:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T15:51:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T15:51:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T15:51:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T15:51:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 45 k 2024-09-17T15:51:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 143 k 2024-09-17T15:51:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:19.3.0-4975.g1889a165.el9 ceph 161 k 2024-09-17T15:51:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T15:51:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:19.3.0-4975.g1889a165.el9 ceph 321 k 2024-09-17T15:51:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 305 k 2024-09-17T15:51:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 99 k 2024-09-17T15:51:05.188 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T15:51:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T15:51:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T15:51:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T15:51:05.189 INFO:teuthology.orchestra.run.smithi162.stdout:Installing weak dependencies: 2024-09-17T15:51:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T15:51:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:05.189 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:51:05.189 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:51:05.189 INFO:teuthology.orchestra.run.smithi162.stdout:Install 31 Packages 2024-09-17T15:51:05.189 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 2 Packages 2024-09-17T15:51:05.189 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:05.190 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 62 M 2024-09-17T15:51:05.190 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:51:06.215 INFO:teuthology.orchestra.run.smithi162.stdout:(1/33): ceph-base-19.3.0-4975.g1889a165.el9.x86 10 MB/s | 5.5 MB 00:00 2024-09-17T15:51:06.281 INFO:teuthology.orchestra.run.smithi162.stdout:(2/33): ceph-selinux-19.3.0-4975.g1889a165.el9. 379 kB/s | 25 kB 00:00 2024-09-17T15:51:06.382 INFO:teuthology.orchestra.run.smithi162.stdout:(3/33): libcephfs2-19.3.0-4975.g1889a165.el9.x8 7.4 MB/s | 756 kB 00:00 2024-09-17T15:51:06.523 INFO:teuthology.orchestra.run.smithi162.stdout:(4/33): ceph-radosgw-19.3.0-4975.g1889a165.el9. 13 MB/s | 11 MB 00:00 2024-09-17T15:51:06.548 INFO:teuthology.orchestra.run.smithi162.stdout:(5/33): libradosstriper1-19.3.0-4975.g1889a165. 3.0 MB/s | 508 kB 00:00 2024-09-17T15:51:06.599 INFO:teuthology.orchestra.run.smithi162.stdout:(6/33): python3-ceph-argparse-19.3.0-4975.g1889 903 kB/s | 45 kB 00:00 2024-09-17T15:51:06.849 INFO:teuthology.orchestra.run.smithi162.stdout:(7/33): ceph-common-19.3.0-4975.g1889a165.el9.x 19 MB/s | 22 MB 00:01 2024-09-17T15:51:06.874 INFO:teuthology.orchestra.run.smithi162.stdout:(8/33): python3-ceph-common-19.3.0-4975.g1889a1 520 kB/s | 143 kB 00:00 2024-09-17T15:51:06.900 INFO:teuthology.orchestra.run.smithi162.stdout:(9/33): python3-cephfs-19.3.0-4975.g1889a165.el 3.1 MB/s | 161 kB 00:00 2024-09-17T15:51:06.925 INFO:teuthology.orchestra.run.smithi162.stdout:(10/33): python3-rados-19.3.0-4975.g1889a165.el 6.2 MB/s | 321 kB 00:00 2024-09-17T15:51:06.950 INFO:teuthology.orchestra.run.smithi162.stdout:(11/33): python3-rbd-19.3.0-4975.g1889a165.el9. 6.0 MB/s | 305 kB 00:00 2024-09-17T15:51:06.975 INFO:teuthology.orchestra.run.smithi162.stdout:(12/33): python3-rgw-19.3.0-4975.g1889a165.el9. 1.9 MB/s | 99 kB 00:00 2024-09-17T15:51:07.051 INFO:teuthology.orchestra.run.smithi162.stdout:(13/33): librgw2-19.3.0-4975.g1889a165.el9.x86_ 10 MB/s | 5.5 MB 00:00 2024-09-17T15:51:07.109 INFO:teuthology.orchestra.run.smithi162.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 672 kB/s | 107 kB 00:00 2024-09-17T15:51:07.151 INFO:teuthology.orchestra.run.smithi162.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-17T15:51:07.210 INFO:teuthology.orchestra.run.smithi162.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 728 kB/s | 171 kB 00:00 2024-09-17T15:51:07.235 INFO:teuthology.orchestra.run.smithi162.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 870 kB/s | 160 kB 00:00 2024-09-17T15:51:07.277 INFO:teuthology.orchestra.run.smithi162.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-17T15:51:07.319 INFO:teuthology.orchestra.run.smithi162.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-17T15:51:07.369 INFO:teuthology.orchestra.run.smithi162.stdout:(20/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.4 MB/s | 221 kB 00:00 2024-09-17T15:51:07.395 INFO:teuthology.orchestra.run.smithi162.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 478 kB/s | 36 kB 00:00 2024-09-17T15:51:07.428 INFO:teuthology.orchestra.run.smithi162.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 810 kB/s | 47 kB 00:00 2024-09-17T15:51:07.462 INFO:teuthology.orchestra.run.smithi162.stdout:(23/33): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-09-17T15:51:07.512 INFO:teuthology.orchestra.run.smithi162.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-17T15:51:07.538 INFO:teuthology.orchestra.run.smithi162.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2024-09-17T15:51:07.562 INFO:teuthology.orchestra.run.smithi162.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 279 kB/s | 26 kB 00:00 2024-09-17T15:51:07.590 INFO:teuthology.orchestra.run.smithi162.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-17T15:51:07.624 INFO:teuthology.orchestra.run.smithi162.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-17T15:51:07.674 INFO:teuthology.orchestra.run.smithi162.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.4 MB/s | 839 kB 00:00 2024-09-17T15:51:07.733 INFO:teuthology.orchestra.run.smithi162.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-17T15:51:07.808 INFO:teuthology.orchestra.run.smithi162.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-17T15:51:07.967 INFO:teuthology.orchestra.run.smithi162.stdout:(32/33): librados2-19.3.0-4975.g1889a165.el9.x8 12 MB/s | 3.5 MB 00:00 2024-09-17T15:51:08.033 INFO:teuthology.orchestra.run.smithi162.stdout:(33/33): librbd1-19.3.0-4975.g1889a165.el9.x86_ 11 MB/s | 3.3 MB 00:00 2024-09-17T15:51:08.039 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:51:08.039 INFO:teuthology.orchestra.run.smithi162.stdout:Total 22 MB/s | 62 MB 00:02 2024-09-17T15:51:08.152 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:51:08.190 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:51:08.190 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:51:08.844 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:51:08.844 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:51:10.424 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:51:10.518 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-17T15:51:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-17T15:51:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-17T15:51:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T15:51:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T15:51:10.814 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T15:51:10.867 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T15:51:11.260 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-17T15:51:11.341 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:51:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:51:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 8/35 2024-09-17T15:51:11.562 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T15:51:11.622 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T15:51:11.659 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-17T15:51:11.703 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 11/35 2024-09-17T15:51:11.782 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T15:51:11.820 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T15:51:11.884 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T15:51:11.927 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-17T15:51:12.038 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-17T15:51:12.070 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-17T15:51:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-17T15:51:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-17T15:51:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-17T15:51:12.206 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T15:51:12.243 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-17T15:51:12.564 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-17T15:51:12.619 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T15:51:12.711 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T15:51:12.749 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 24/35 2024-09-17T15:51:12.794 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-17T15:51:13.285 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-17T15:51:13.314 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T15:51:13.385 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T15:51:13.464 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 28/35 2024-09-17T15:51:13.500 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 29/35 2024-09-17T15:51:15.785 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:51:15.821 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:51:16.393 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:51:16.433 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T15:51:16.547 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T15:51:16.548 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T15:51:16.548 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T15:51:16.548 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:16.593 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:51:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:51:26.785 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2024-09-17T15:51:26.786 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2024-09-17T15:51:26.786 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2024-09-17T15:51:26.786 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2024-09-17T15:51:26.786 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2024-09-17T15:51:26.786 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /root 2024-09-17T15:51:26.786 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2024-09-17T15:51:26.786 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:27.809 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T15:51:27.834 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T15:51:27.834 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:51:27.834 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T15:51:27.834 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:51:27.834 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:51:27.834 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:27.836 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T15:51:27.921 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T15:51:27.921 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:51:29.197 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:51:29.197 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 1/35 2024-09-17T15:51:29.198 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2/35 2024-09-17T15:51:29.198 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 3/35 2024-09-17T15:51:29.198 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 4/35 2024-09-17T15:51:29.198 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 5/35 2024-09-17T15:51:29.198 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 6/35 2024-09-17T15:51:29.198 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:51:29.198 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 8/35 2024-09-17T15:51:29.198 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 9/35 2024-09-17T15:51:29.198 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 10/35 2024-09-17T15:51:29.198 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 11/35 2024-09-17T15:51:29.198 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T15:51:29.198 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 13/35 2024-09-17T15:51:29.198 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-17T15:51:29.198 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-17T15:51:29.199 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-17T15:51:29.199 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-17T15:51:29.199 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-17T15:51:29.199 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-17T15:51:29.199 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T15:51:29.199 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-17T15:51:29.199 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-17T15:51:29.199 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-17T15:51:29.199 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-17T15:51:29.199 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-17T15:51:29.199 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-17T15:51:29.199 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-17T15:51:29.200 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-17T15:51:29.200 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-17T15:51:29.200 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-17T15:51:29.201 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-17T15:51:29.201 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:51:29.201 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-17T15:51:29.201 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 34/35 2024-09-17T15:51:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:51:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:30.219 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2024-09-17T15:51:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:30.219 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:51:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T15:51:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T15:51:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T15:51:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T15:51:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-17T15:51:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T15:51:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T15:51:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T15:51:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T15:51:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T15:51:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T15:51:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T15:51:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T15:51:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T15:51:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T15:51:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T15:51:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T15:51:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T15:51:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:30.222 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:51:30.590 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-test 2024-09-17T15:51:31.174 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:29 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:51:31.312 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:51:31.312 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:51:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-17T15:51:31.313 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:51:31.313 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:51:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:19.3.0-4975.g1889a165.el9 ceph 46 M 2024-09-17T15:51:31.313 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-17T15:51:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:19.3.0-4975.g1889a165.el9 ceph 162 k 2024-09-17T15:51:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T15:51:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T15:51:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:31.313 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:51:31.313 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:51:31.313 INFO:teuthology.orchestra.run.smithi162.stdout:Install 4 Packages 2024-09-17T15:51:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:31.314 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 46 M 2024-09-17T15:51:31.314 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 201 M 2024-09-17T15:51:31.314 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:51:31.793 INFO:teuthology.orchestra.run.smithi162.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-17T15:51:31.818 INFO:teuthology.orchestra.run.smithi162.stdout:(2/4): libcephsqlite-19.3.0-4975.g1889a165.el9. 725 kB/s | 162 kB 00:00 2024-09-17T15:51:31.843 INFO:teuthology.orchestra.run.smithi162.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-17T15:51:33.043 INFO:teuthology.orchestra.run.smithi162.stdout:(4/4): ceph-test-19.3.0-4975.g1889a165.el9.x86_ 31 MB/s | 46 MB 00:01 2024-09-17T15:51:33.045 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:51:33.046 INFO:teuthology.orchestra.run.smithi162.stdout:Total 27 MB/s | 46 MB 00:01 2024-09-17T15:51:33.074 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:51:33.095 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:51:33.095 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:51:33.466 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:51:33.467 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:51:33.982 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:51:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T15:51:34.127 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T15:51:34.154 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T15:51:38.553 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T15:51:38.635 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T15:51:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T15:51:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 1/4 2024-09-17T15:51:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2/4 2024-09-17T15:51:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T15:51:41.311 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T15:51:41.312 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:41.312 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:51:41.312 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:41.312 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:41.312 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T15:51:41.312 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T15:51:41.312 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:41.312 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:51:41.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph 2024-09-17T15:51:42.289 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:40 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:51:42.438 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:51:42.441 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:51:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-17T15:51:42.441 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:51:42.441 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:51:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:19.3.0-4975.g1889a165.el9 ceph 6.6 k 2024-09-17T15:51:42.441 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-17T15:51:42.441 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:19.3.0-4975.g1889a165.el9 ceph 2.4 M 2024-09-17T15:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:19.3.0-4975.g1889a165.el9 ceph 1.5 M 2024-09-17T15:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 291 k 2024-09-17T15:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:19.3.0-4975.g1889a165.el9 ceph 4.8 M 2024-09-17T15:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 17 M 2024-09-17T15:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T15:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T15:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T15:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T15:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T15:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T15:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T15:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T15:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T15:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T15:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections 2024-09-17T15:51:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-17T15:51:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T15:51:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T15:51:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T15:51:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T15:51:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T15:51:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T15:51:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T15:51:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T15:51:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T15:51:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T15:51:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T15:51:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T15:51:42.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T15:51:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T15:51:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T15:51:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T15:51:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T15:51:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T15:51:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T15:51:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T15:51:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:42.444 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:51:42.444 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:51:42.444 INFO:teuthology.orchestra.run.smithi162.stdout:Install 37 Packages 2024-09-17T15:51:42.444 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:42.445 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 31 M 2024-09-17T15:51:42.445 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 109 M 2024-09-17T15:51:42.445 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:51:43.456 INFO:teuthology.orchestra.run.smithi162.stdout:(1/37): ceph-19.3.0-4975.g1889a165.el9.x86_64.r 46 kB/s | 6.6 kB 00:00 2024-09-17T15:51:43.612 INFO:teuthology.orchestra.run.smithi162.stdout:(2/37): ceph-mgr-19.3.0-4975.g1889a165.el9.x86_ 5.1 MB/s | 1.5 MB 00:00 2024-09-17T15:51:43.829 INFO:teuthology.orchestra.run.smithi162.stdout:(3/37): ceph-mon-19.3.0-4975.g1889a165.el9.x86_ 13 MB/s | 4.8 MB 00:00 2024-09-17T15:51:43.896 INFO:teuthology.orchestra.run.smithi162.stdout:(4/37): ceph-mds-19.3.0-4975.g1889a165.el9.x86_ 4.1 MB/s | 2.4 MB 00:00 2024-09-17T15:51:43.922 INFO:teuthology.orchestra.run.smithi162.stdout:(5/37): ceph-mgr-modules-core-19.3.0-4975.g1889 3.1 MB/s | 291 kB 00:00 2024-09-17T15:51:44.247 INFO:teuthology.orchestra.run.smithi162.stdout:(6/37): ceph-osd-19.3.0-4975.g1889a165.el9.x86_ 27 MB/s | 17 MB 00:00 2024-09-17T15:51:44.272 INFO:teuthology.orchestra.run.smithi162.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 674 kB/s | 253 kB 00:00 2024-09-17T15:51:44.314 INFO:teuthology.orchestra.run.smithi162.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-17T15:51:44.356 INFO:teuthology.orchestra.run.smithi162.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-17T15:51:44.385 INFO:teuthology.orchestra.run.smithi162.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 774 kB/s | 106 kB 00:00 2024-09-17T15:51:44.419 INFO:teuthology.orchestra.run.smithi162.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 2.5 MB/s | 1.2 MB 00:00 2024-09-17T15:51:44.444 INFO:teuthology.orchestra.run.smithi162.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.4 MB/s | 215 kB 00:00 2024-09-17T15:51:44.478 INFO:teuthology.orchestra.run.smithi162.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-09-17T15:51:44.494 INFO:teuthology.orchestra.run.smithi162.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 692 kB/s | 35 kB 00:00 2024-09-17T15:51:44.520 INFO:teuthology.orchestra.run.smithi162.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-17T15:51:44.545 INFO:teuthology.orchestra.run.smithi162.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-17T15:51:44.571 INFO:teuthology.orchestra.run.smithi162.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 671 kB/s | 51 kB 00:00 2024-09-17T15:51:44.596 INFO:teuthology.orchestra.run.smithi162.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 550 kB/s | 42 kB 00:00 2024-09-17T15:51:44.638 INFO:teuthology.orchestra.run.smithi162.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-17T15:51:44.663 INFO:teuthology.orchestra.run.smithi162.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 7.2 MB/s | 172 kB 00:00 2024-09-17T15:51:44.689 INFO:teuthology.orchestra.run.smithi162.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 470 kB/s | 43 kB 00:00 2024-09-17T15:51:44.714 INFO:teuthology.orchestra.run.smithi162.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 429 kB/s | 11 kB 00:00 2024-09-17T15:51:44.739 INFO:teuthology.orchestra.run.smithi162.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 709 kB/s | 18 kB 00:00 2024-09-17T15:51:44.765 INFO:teuthology.orchestra.run.smithi162.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-09-17T15:51:44.790 INFO:teuthology.orchestra.run.smithi162.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 458 kB/s | 23 kB 00:00 2024-09-17T15:51:44.823 INFO:teuthology.orchestra.run.smithi162.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 332 kB/s | 19 kB 00:00 2024-09-17T15:51:44.849 INFO:teuthology.orchestra.run.smithi162.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 380 kB/s | 20 kB 00:00 2024-09-17T15:51:44.874 INFO:teuthology.orchestra.run.smithi162.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-17T15:51:44.899 INFO:teuthology.orchestra.run.smithi162.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-17T15:51:44.925 INFO:teuthology.orchestra.run.smithi162.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 657 kB/s | 16 kB 00:00 2024-09-17T15:51:44.950 INFO:teuthology.orchestra.run.smithi162.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-17T15:51:44.975 INFO:teuthology.orchestra.run.smithi162.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-17T15:51:45.001 INFO:teuthology.orchestra.run.smithi162.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-09-17T15:51:45.026 INFO:teuthology.orchestra.run.smithi162.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-17T15:51:45.051 INFO:teuthology.orchestra.run.smithi162.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 800 kB/s | 20 kB 00:00 2024-09-17T15:51:45.076 INFO:teuthology.orchestra.run.smithi162.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-09-17T15:51:45.118 INFO:teuthology.orchestra.run.smithi162.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-09-17T15:51:45.122 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:51:45.122 INFO:teuthology.orchestra.run.smithi162.stdout:Total 11 MB/s | 31 MB 00:02 2024-09-17T15:51:45.269 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:51:45.311 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:51:45.311 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:51:45.779 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:51:45.780 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:51:47.177 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:51:47.246 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T15:51:47.282 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T15:51:47.316 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T15:51:47.397 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T15:51:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T15:51:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T15:51:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T15:51:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T15:51:47.692 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T15:51:47.728 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T15:51:47.757 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T15:51:47.792 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T15:51:47.834 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T15:51:47.884 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T15:51:47.926 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T15:51:47.961 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T15:51:48.006 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T15:51:48.047 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T15:51:48.120 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T15:51:48.276 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T15:51:48.355 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T15:51:48.409 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T15:51:48.447 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T15:51:48.504 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T15:51:48.560 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T15:51:48.747 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T15:51:48.793 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T15:51:48.871 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T15:51:48.960 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T15:51:48.992 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T15:51:49.122 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T15:51:49.290 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 32/37 2024-09-17T15:51:49.320 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T15:51:49.346 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T15:51:49.346 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:51:49.346 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T15:51:49.347 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T15:51:49.347 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T15:51:49.347 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:50.812 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T15:51:50.840 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T15:51:50.840 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:51:50.840 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T15:51:50.840 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T15:51:50.840 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T15:51:50.840 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T15:51:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T15:51:51.321 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:51:51.321 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T15:51:51.321 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T15:51:51.321 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T15:51:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:51.548 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T15:51:51.571 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T15:51:51.571 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:51:51.571 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T15:51:51.571 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T15:51:51.571 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T15:51:51.571 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:51.635 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T15:51:53.366 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T15:51:53.366 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 1/37 2024-09-17T15:51:53.366 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2/37 2024-09-17T15:51:53.366 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 3/37 2024-09-17T15:51:53.367 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 4/37 2024-09-17T15:51:53.367 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 5/37 2024-09-17T15:51:53.367 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 6/37 2024-09-17T15:51:53.367 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T15:51:53.367 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T15:51:53.367 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T15:51:53.367 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T15:51:53.367 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T15:51:53.367 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T15:51:53.367 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T15:51:53.368 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T15:51:53.368 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T15:51:53.368 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T15:51:53.368 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T15:51:53.368 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T15:51:53.368 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T15:51:53.368 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T15:51:53.368 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T15:51:53.368 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T15:51:53.368 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T15:51:53.368 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T15:51:53.368 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T15:51:53.368 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T15:51:53.368 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T15:51:53.369 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T15:51:53.369 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T15:51:53.369 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T15:51:53.369 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T15:51:53.369 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T15:51:53.369 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T15:51:53.369 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T15:51:53.369 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T15:51:53.369 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T15:51:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T15:51:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:54.269 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:51:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:51:54.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-base 2024-09-17T15:51:55.263 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:51:55.352 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:51:55.394 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:51:55.395 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-17T15:51:55.395 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:51:55.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install cephadm 2024-09-17T15:51:56.065 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:54 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:51:56.197 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:51:56.198 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:51:56.198 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-17T15:51:56.198 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:51:56.198 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:51:56.198 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 784 k 2024-09-17T15:51:56.198 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:56.200 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:51:56.200 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:51:56.200 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-17T15:51:56.200 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:56.200 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 784 k 2024-09-17T15:51:56.201 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 791 k 2024-09-17T15:51:56.201 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:51:56.441 INFO:teuthology.orchestra.run.smithi162.stdout:cephadm-19.3.0-4975.g1889a165.el9.noarch.rpm 3.2 MB/s | 784 kB 00:00 2024-09-17T15:51:56.442 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:51:56.442 INFO:teuthology.orchestra.run.smithi162.stdout:Total 3.1 MB/s | 784 kB 00:00 2024-09-17T15:51:56.442 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:51:56.447 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:51:56.447 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:51:56.453 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:51:56.453 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:51:56.524 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:51:57.002 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:51:57.105 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:51:57.468 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:51:57.718 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:51:57.718 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:57.718 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:51:57.718 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:51:57.719 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:57.719 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:51:57.955 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T15:51:58.549 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:56 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:51:58.687 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:51:58.687 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:51:58.687 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-09-17T15:51:58.688 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:51:58.688 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:51:58.688 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4975.g1889a165.el9 ceph 145 k 2024-09-17T15:51:58.688 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:58.688 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:51:58.688 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:51:58.688 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-17T15:51:58.688 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:51:58.688 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 145 k 2024-09-17T15:51:58.688 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 432 k 2024-09-17T15:51:58.688 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:51:58.982 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-immutable-object-cache-19.3.0-4975.g1889a1 495 kB/s | 145 kB 00:00 2024-09-17T15:51:58.982 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:51:58.982 INFO:teuthology.orchestra.run.smithi162.stdout:Total 492 kB/s | 145 kB 00:00 2024-09-17T15:51:58.983 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:51:58.990 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:51:58.990 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:51:59.046 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:51:59.046 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:51:59.192 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:51:59.266 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:51:59.291 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:51:59.291 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:51:59.291 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T15:51:59.291 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:00.205 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:52:00.206 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:00.206 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:52:00.206 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:52:00.206 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:00.206 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:00.444 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr 2024-09-17T15:52:01.039 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:01.132 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:52:01.176 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:01.178 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-17T15:52:01.178 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:01.266 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T15:52:01.838 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:01.974 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:01.975 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:01.975 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-17T15:52:01.975 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:01.975 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:52:01.976 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 14 M 2024-09-17T15:52:01.976 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-17T15:52:01.976 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T15:52:01.976 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 34 k 2024-09-17T15:52:01.976 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 16 k 2024-09-17T15:52:01.976 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T15:52:01.976 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T15:52:01.976 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T15:52:01.976 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T15:52:01.976 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T15:52:01.976 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T15:52:01.976 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T15:52:01.976 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T15:52:01.976 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:01.976 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:52:01.977 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:01.977 INFO:teuthology.orchestra.run.smithi162.stdout:Install 12 Packages 2024-09-17T15:52:01.977 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:01.977 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 19 M 2024-09-17T15:52:01.977 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 159 M 2024-09-17T15:52:01.977 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:52:02.438 INFO:teuthology.orchestra.run.smithi162.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4975.g18 240 kB/s | 34 kB 00:00 2024-09-17T15:52:02.455 INFO:teuthology.orchestra.run.smithi162.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4975.g188 103 kB/s | 16 kB 00:00 2024-09-17T15:52:02.680 INFO:teuthology.orchestra.run.smithi162.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-17T15:52:02.839 INFO:teuthology.orchestra.run.smithi162.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.5 MB/s | 1.0 MB 00:00 2024-09-17T15:52:03.015 INFO:teuthology.orchestra.run.smithi162.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4975.g1889a16 20 MB/s | 14 MB 00:00 2024-09-17T15:52:03.065 INFO:teuthology.orchestra.run.smithi162.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.4 MB/s | 548 kB 00:00 2024-09-17T15:52:03.081 INFO:teuthology.orchestra.run.smithi162.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 300 kB/s | 19 kB 00:00 2024-09-17T15:52:03.123 INFO:teuthology.orchestra.run.smithi162.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.4 MB/s | 144 kB 00:00 2024-09-17T15:52:03.157 INFO:teuthology.orchestra.run.smithi162.stdout:(9/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.8 MB/s | 863 kB 00:00 2024-09-17T15:52:03.174 INFO:teuthology.orchestra.run.smithi162.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 610 kB/s | 31 kB 00:00 2024-09-17T15:52:03.224 INFO:teuthology.orchestra.run.smithi162.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 13 MB/s | 2.0 MB 00:00 2024-09-17T15:52:03.266 INFO:teuthology.orchestra.run.smithi162.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-09-17T15:52:03.270 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:52:03.270 INFO:teuthology.orchestra.run.smithi162.stdout:Total 15 MB/s | 19 MB 00:01 2024-09-17T15:52:03.319 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:52:03.337 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:52:03.337 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:52:03.607 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:52:03.608 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:52:04.219 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:52:04.308 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-17T15:52:04.360 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-17T15:52:04.415 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-17T15:52:04.446 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-17T15:52:04.550 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-17T15:52:04.689 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-17T15:52:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-17T15:52:04.843 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-17T15:52:04.872 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-17T15:52:04.904 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 10/12 2024-09-17T15:52:08.085 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 11/12 2024-09-17T15:52:08.142 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T15:52:09.980 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T15:52:09.980 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 1/12 2024-09-17T15:52:09.980 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 2/12 2024-09-17T15:52:09.980 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 3/12 2024-09-17T15:52:09.980 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-17T15:52:09.980 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-17T15:52:09.981 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-17T15:52:09.981 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-17T15:52:09.981 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-17T15:52:09.981 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-17T15:52:09.981 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-17T15:52:09.981 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-17T15:52:10.312 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-17T15:52:10.312 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:10.312 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:52:10.312 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T15:52:10.312 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:52:10.312 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:52:10.312 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:52:10.312 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T15:52:10.312 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T15:52:10.313 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T15:52:10.313 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T15:52:10.313 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T15:52:10.313 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T15:52:10.313 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T15:52:10.313 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T15:52:10.313 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:10.313 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:10.616 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T15:52:11.201 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:09 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:11.341 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:11.342 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-09-17T15:52:11.343 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-17T15:52:11.343 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-09-17T15:52:11.343 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:52:11.343 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 7.4 M 2024-09-17T15:52:11.343 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-17T15:52:11.343 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T15:52:11.343 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T15:52:11.343 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T15:52:11.343 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T15:52:11.344 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T15:52:11.344 INFO:teuthology.orchestra.run.smithi162.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T15:52:11.344 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T15:52:11.344 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T15:52:11.344 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T15:52:11.344 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T15:52:11.344 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T15:52:11.344 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:11.344 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:52:11.344 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-09-17T15:52:11.344 INFO:teuthology.orchestra.run.smithi162.stdout:Install 12 Packages 2024-09-17T15:52:11.344 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:11.345 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 38 M 2024-09-17T15:52:11.345 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 212 M 2024-09-17T15:52:11.345 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:52:11.800 INFO:teuthology.orchestra.run.smithi162.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.5 MB/s | 191 kB 00:00 2024-09-17T15:52:11.834 INFO:teuthology.orchestra.run.smithi162.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.9 MB/s | 801 kB 00:00 2024-09-17T15:52:11.909 INFO:teuthology.orchestra.run.smithi162.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 301 kB/s | 33 kB 00:00 2024-09-17T15:52:11.959 INFO:teuthology.orchestra.run.smithi162.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-17T15:52:12.010 INFO:teuthology.orchestra.run.smithi162.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 769 kB/s | 38 kB 00:00 2024-09-17T15:52:12.143 INFO:teuthology.orchestra.run.smithi162.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-49 16 MB/s | 7.4 MB 00:00 2024-09-17T15:52:12.344 INFO:teuthology.orchestra.run.smithi162.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-09-17T15:52:12.711 INFO:teuthology.orchestra.run.smithi162.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.4 MB/s | 3.0 MB 00:00 2024-09-17T15:52:12.795 INFO:teuthology.orchestra.run.smithi162.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-09-17T15:52:13.036 INFO:teuthology.orchestra.run.smithi162.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.8 MB/s | 4.9 MB 00:01 2024-09-17T15:52:13.378 INFO:teuthology.orchestra.run.smithi162.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.9 MB/s | 5.1 MB 00:01 2024-09-17T15:52:14.011 INFO:teuthology.orchestra.run.smithi162.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-09-17T15:52:14.012 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:52:14.013 INFO:teuthology.orchestra.run.smithi162.stdout:Total 14 MB/s | 38 MB 00:02 2024-09-17T15:52:14.367 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:52:14.393 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:52:14.393 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:52:14.825 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:52:14.825 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:52:15.593 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:52:15.656 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T15:52:15.702 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T15:52:15.805 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T15:52:15.845 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T15:52:16.032 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T15:52:16.069 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T15:52:16.154 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T15:52:16.829 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T15:52:16.950 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T15:52:18.220 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T15:52:19.267 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T15:52:19.295 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T15:52:21.953 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T15:52:21.954 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 1/12 2024-09-17T15:52:21.954 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T15:52:21.954 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T15:52:21.954 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T15:52:21.954 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T15:52:21.954 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T15:52:21.954 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T15:52:21.954 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T15:52:21.954 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T15:52:21.954 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T15:52:21.954 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T15:52:22.359 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T15:52:22.359 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:22.359 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:52:22.359 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:52:22.359 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T15:52:22.359 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T15:52:22.359 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T15:52:22.359 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T15:52:22.359 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T15:52:22.360 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T15:52:22.360 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T15:52:22.360 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T15:52:22.360 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T15:52:22.360 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T15:52:22.360 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T15:52:22.360 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:22.360 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:22.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-rook 2024-09-17T15:52:23.368 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:21 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:23.502 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:23.504 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:23.504 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-17T15:52:23.504 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:23.504 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:52:23.504 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 49 k 2024-09-17T15:52:23.504 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-17T15:52:23.504 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T15:52:23.504 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T15:52:23.504 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T15:52:23.504 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T15:52:23.504 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T15:52:23.504 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T15:52:23.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T15:52:23.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T15:52:23.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T15:52:23.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T15:52:23.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T15:52:23.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T15:52:23.505 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:23.505 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:52:23.505 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:23.505 INFO:teuthology.orchestra.run.smithi162.stdout:Install 13 Packages 2024-09-17T15:52:23.505 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:23.506 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 2.2 M 2024-09-17T15:52:23.506 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 26 M 2024-09-17T15:52:23.506 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:52:23.930 INFO:teuthology.orchestra.run.smithi162.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 164 kB/s | 19 kB 00:00 2024-09-17T15:52:23.955 INFO:teuthology.orchestra.run.smithi162.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 180 kB/s | 26 kB 00:00 2024-09-17T15:52:23.980 INFO:teuthology.orchestra.run.smithi162.stdout:(3/13): ceph-mgr-rook-19.3.0-4975.g1889a165.el9 294 kB/s | 49 kB 00:00 2024-09-17T15:52:24.064 INFO:teuthology.orchestra.run.smithi162.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-17T15:52:24.089 INFO:teuthology.orchestra.run.smithi162.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-17T15:52:24.123 INFO:teuthology.orchestra.run.smithi162.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 911 kB/s | 54 kB 00:00 2024-09-17T15:52:24.157 INFO:teuthology.orchestra.run.smithi162.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 479 kB/s | 32 kB 00:00 2024-09-17T15:52:24.182 INFO:teuthology.orchestra.run.smithi162.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 249 kB/s | 14 kB 00:00 2024-09-17T15:52:24.215 INFO:teuthology.orchestra.run.smithi162.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-17T15:52:24.249 INFO:teuthology.orchestra.run.smithi162.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.4 MB/s | 220 kB 00:00 2024-09-17T15:52:24.275 INFO:teuthology.orchestra.run.smithi162.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-17T15:52:24.300 INFO:teuthology.orchestra.run.smithi162.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 704 kB/s | 59 kB 00:00 2024-09-17T15:52:24.334 INFO:teuthology.orchestra.run.smithi162.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-17T15:52:24.336 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:52:24.336 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.7 MB/s | 2.2 MB 00:00 2024-09-17T15:52:24.400 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:52:24.415 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:52:24.415 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:52:24.503 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:52:24.503 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:52:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:52:25.166 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T15:52:25.226 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T15:52:25.291 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T15:52:25.328 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T15:52:25.378 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T15:52:25.487 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T15:52:25.590 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T15:52:25.646 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T15:52:26.372 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T15:52:26.427 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T15:52:26.461 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T15:52:26.509 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T15:52:26.531 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T15:52:27.279 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T15:52:27.280 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 1/13 2024-09-17T15:52:27.280 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T15:52:27.280 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T15:52:27.280 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T15:52:27.280 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T15:52:27.280 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T15:52:27.280 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T15:52:27.280 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T15:52:27.280 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T15:52:27.280 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T15:52:27.280 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T15:52:27.280 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T15:52:27.606 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T15:52:27.607 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:27.607 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:52:27.607 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:52:27.607 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T15:52:27.607 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T15:52:27.607 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T15:52:27.607 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T15:52:27.607 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T15:52:27.607 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T15:52:27.607 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T15:52:27.607 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T15:52:27.607 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T15:52:27.607 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T15:52:27.607 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T15:52:27.607 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T15:52:27.608 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:27.608 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:27.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T15:52:28.478 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:26 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:28.612 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:28.613 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-17T15:52:28.613 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:28.613 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:52:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 154 k 2024-09-17T15:52:28.613 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-17T15:52:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T15:52:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T15:52:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T15:52:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T15:52:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T15:52:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:28.614 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:52:28.614 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:28.614 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2024-09-17T15:52:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:28.615 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 7.0 M 2024-09-17T15:52:28.615 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 33 M 2024-09-17T15:52:28.615 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:52:29.171 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-17T15:52:29.221 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-17T15:52:29.246 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-17T15:52:29.272 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-17T15:52:29.372 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-09-17T15:52:29.455 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): ceph-mgr-cephadm-19.3.0-4975.g1889a165.e 361 kB/s | 154 kB 00:00 2024-09-17T15:52:29.457 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:52:29.457 INFO:teuthology.orchestra.run.smithi162.stdout:Total 8.3 MB/s | 7.0 MB 00:00 2024-09-17T15:52:29.528 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:52:29.539 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:52:29.539 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:52:29.640 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:52:29.641 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:52:29.954 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:52:30.139 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T15:52:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T15:52:30.724 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T15:52:30.819 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T15:52:30.895 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T15:52:30.920 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T15:52:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T15:52:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/6 2024-09-17T15:52:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T15:52:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T15:52:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T15:52:31.712 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T15:52:31.981 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T15:52:31.981 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:31.981 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:52:31.982 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:52:31.982 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T15:52:31.982 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T15:52:31.982 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T15:52:31.982 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T15:52:31.982 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T15:52:31.982 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:31.982 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:32.239 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-fuse 2024-09-17T15:52:32.816 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:30 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:32.949 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:32.949 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:32.949 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-17T15:52:32.949 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:32.949 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:52:32.950 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 899 k 2024-09-17T15:52:32.950 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-17T15:52:32.950 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T15:52:32.950 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:32.950 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:52:32.950 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:32.950 INFO:teuthology.orchestra.run.smithi162.stdout:Install 2 Packages 2024-09-17T15:52:32.950 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:32.951 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 979 k 2024-09-17T15:52:32.951 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 2.9 M 2024-09-17T15:52:32.951 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:52:33.683 INFO:teuthology.orchestra.run.smithi162.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 126 kB/s | 80 kB 00:00 2024-09-17T15:52:33.716 INFO:teuthology.orchestra.run.smithi162.stdout:(2/2): ceph-fuse-19.3.0-4975.g1889a165.el9.x86_ 1.3 MB/s | 899 kB 00:00 2024-09-17T15:52:33.717 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:52:33.718 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.2 MB/s | 979 kB 00:00 2024-09-17T15:52:33.731 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:52:33.756 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:52:33.756 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:52:33.842 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:52:33.842 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:52:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:52:34.178 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T15:52:34.250 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T15:52:34.979 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T15:52:34.979 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/2 2024-09-17T15:52:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T15:52:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:35.256 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:52:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T15:52:35.257 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:35.257 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:35.572 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-volume 2024-09-17T15:52:36.165 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:34 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:36.301 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:36.302 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-17T15:52:36.302 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:36.302 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:52:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 285 k 2024-09-17T15:52:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:36.302 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:52:36.302 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:36.302 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-17T15:52:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:36.302 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 285 k 2024-09-17T15:52:36.302 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 1.3 M 2024-09-17T15:52:36.303 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:52:36.508 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-volume-19.3.0-4975.g1889a165.el9.noarch.rp 1.4 MB/s | 285 kB 00:00 2024-09-17T15:52:36.508 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:52:36.508 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.4 MB/s | 285 kB 00:00 2024-09-17T15:52:36.509 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:52:36.515 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:52:36.515 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:52:36.543 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:52:36.543 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:52:36.764 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:52:36.824 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:52:36.841 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:52:36.841 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:52:36.841 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T15:52:36.841 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:37.847 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:52:37.847 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:37.847 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:52:37.847 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:52:37.847 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:37.847 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:38.153 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados-devel 2024-09-17T15:52:38.752 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:36 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:38.890 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:38.890 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-09-17T15:52:38.890 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:38.890 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:52:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 126 k 2024-09-17T15:52:38.890 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:38.891 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:52:38.891 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:38.891 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-17T15:52:38.891 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:38.891 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 126 k 2024-09-17T15:52:38.891 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 456 k 2024-09-17T15:52:38.891 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:52:39.137 INFO:teuthology.orchestra.run.smithi162.stdout:librados-devel-19.3.0-4975.g1889a165.el9.x86_64 517 kB/s | 126 kB 00:00 2024-09-17T15:52:39.137 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:52:39.138 INFO:teuthology.orchestra.run.smithi162.stdout:Total 513 kB/s | 126 kB 00:00 2024-09-17T15:52:39.138 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:52:39.143 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:52:39.143 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:52:39.207 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:52:39.208 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:52:39.371 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:52:39.477 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:52:39.955 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:52:40.202 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:52:40.203 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:40.203 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:52:40.203 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:52:40.203 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:40.203 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:40.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs2 2024-09-17T15:52:41.070 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:39 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:41.162 INFO:teuthology.orchestra.run.smithi162.stdout:Package libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:52:41.206 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:41.207 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-17T15:52:41.207 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:41.293 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs-devel 2024-09-17T15:52:41.888 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:39 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:42.028 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:42.029 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:42.029 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-09-17T15:52:42.029 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:42.029 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:52:42.030 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 32 k 2024-09-17T15:52:42.030 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:42.030 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:52:42.030 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:42.030 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-17T15:52:42.030 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:42.030 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 32 k 2024-09-17T15:52:42.030 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 142 k 2024-09-17T15:52:42.030 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:52:42.175 INFO:teuthology.orchestra.run.smithi162.stdout:libcephfs-devel-19.3.0-4975.g1889a165.el9.x86_6 220 kB/s | 32 kB 00:00 2024-09-17T15:52:42.176 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:52:42.176 INFO:teuthology.orchestra.run.smithi162.stdout:Total 217 kB/s | 32 kB 00:00 2024-09-17T15:52:42.176 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:52:42.180 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:52:42.180 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:52:42.203 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:52:42.203 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:52:42.311 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:52:42.415 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:52:42.820 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:52:43.044 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:52:43.045 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:43.045 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:52:43.045 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:52:43.045 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:43.045 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:43.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados2 2024-09-17T15:52:43.849 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:41 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:43.939 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:52:43.983 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:43.984 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-17T15:52:43.984 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:44.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librbd1 2024-09-17T15:52:44.655 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:44.745 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:52:44.788 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:44.789 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-17T15:52:44.790 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:44.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rados 2024-09-17T15:52:45.454 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:45.543 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:52:45.588 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:45.589 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-17T15:52:45.589 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:45.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rgw 2024-09-17T15:52:46.252 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:44 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:46.341 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:52:46.384 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:46.385 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-17T15:52:46.385 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:46.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-cephfs 2024-09-17T15:52:47.058 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:45 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:47.147 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:52:47.190 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:47.191 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-17T15:52:47.192 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:47.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rbd 2024-09-17T15:52:47.858 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:45 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:47.948 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:52:47.991 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:47.992 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-17T15:52:47.992 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:48.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-fuse 2024-09-17T15:52:48.658 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:48.791 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:48.792 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:48.792 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-09-17T15:52:48.792 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:48.792 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:52:48.792 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 85 k 2024-09-17T15:52:48.793 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:48.793 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:52:48.793 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:48.793 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-17T15:52:48.793 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:48.793 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 85 k 2024-09-17T15:52:48.793 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 231 k 2024-09-17T15:52:48.793 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:52:48.995 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-fuse-19.3.0-4975.g1889a165.el9.x86_64.rpm 422 kB/s | 85 kB 00:00 2024-09-17T15:52:48.995 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:52:48.996 INFO:teuthology.orchestra.run.smithi162.stdout:Total 419 kB/s | 85 kB 00:00 2024-09-17T15:52:48.996 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:52:49.003 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:52:49.003 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:52:49.066 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:52:49.067 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:52:49.211 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:52:49.300 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:52:49.688 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:52:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:52:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:49.955 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:52:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:52:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:49.955 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:50.291 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-mirror 2024-09-17T15:52:50.894 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:48 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:51.033 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:51.036 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:51.036 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-09-17T15:52:51.036 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:51.036 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:52:51.036 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.2 M 2024-09-17T15:52:51.036 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:51.037 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:52:51.037 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:51.037 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-17T15:52:51.037 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:51.038 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 3.2 M 2024-09-17T15:52:51.038 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 14 M 2024-09-17T15:52:51.038 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:52:51.374 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-mirror-19.3.0-4975.g1889a165.el9.x86_64.rpm 9.4 MB/s | 3.2 MB 00:00 2024-09-17T15:52:51.375 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:52:51.375 INFO:teuthology.orchestra.run.smithi162.stdout:Total 9.4 MB/s | 3.2 MB 00:00 2024-09-17T15:52:51.375 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:52:51.384 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:52:51.384 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:52:51.465 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:52:51.466 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:52:51.910 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:52:51.984 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:52:52.009 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:52:52.009 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:52:52.010 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T15:52:52.010 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T15:52:52.010 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T15:52:52.010 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:53.107 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:52:53.107 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:53.107 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:52:53.108 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:52:53.108 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:53.108 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:53.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-nbd 2024-09-17T15:52:53.962 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:51 ago on Tue 17 Sep 2024 03:51:02 PM UTC. 2024-09-17T15:52:54.097 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-17T15:52:54.097 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:54.098 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-09-17T15:52:54.098 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:54.098 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-17T15:52:54.098 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 170 k 2024-09-17T15:52:54.098 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:54.098 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-17T15:52:54.098 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-17T15:52:54.098 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-17T15:52:54.098 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:54.098 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 170 k 2024-09-17T15:52:54.098 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 490 k 2024-09-17T15:52:54.098 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-17T15:52:54.312 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-nbd-19.3.0-4975.g1889a165.el9.x86_64.rpm 797 kB/s | 170 kB 00:00 2024-09-17T15:52:54.313 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:52:54.313 INFO:teuthology.orchestra.run.smithi162.stdout:Total 791 kB/s | 170 kB 00:00 2024-09-17T15:52:54.313 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-17T15:52:54.321 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-17T15:52:54.321 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-17T15:52:54.385 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-17T15:52:54.385 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-17T15:52:54.536 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-17T15:52:54.653 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:52:55.105 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:52:55.359 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:52:55.359 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:55.360 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-17T15:52:55.360 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:52:55.360 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-17T15:52:55.360 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-17T15:52:55.668 DEBUG:teuthology.parallel:result is None 2024-09-17T15:52:55.668 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:52:55.818 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T15:52:55.839 INFO:teuthology.orchestra.run.smithi162.stdout:19.3.0-4975.g1889a165.el9 2024-09-17T15:52:55.840 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4975.g1889a165.el9 2024-09-17T15:52:55.840 INFO:teuthology.task.install:The correct ceph version 19.3.0-4975.g1889a165 is installed. 2024-09-17T15:52:55.842 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T15:52:55.842 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-17T15:52:55.842 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T15:52:55.905 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T15:52:55.906 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-17T15:52:55.906 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T15:52:55.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T15:52:56.035 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T15:52:56.035 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-17T15:52:56.035 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T15:52:56.099 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T15:52:56.163 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T15:52:56.163 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-17T15:52:56.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T15:52:56.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T15:52:56.293 INFO:teuthology.run_tasks:Running task workunit... 2024-09-17T15:52:56.310 INFO:tasks.workunit:Pulling workunits from ref 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:52:56.310 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-17T15:52:56.310 INFO:tasks.workunit:timeout=3h 2024-09-17T15:52:56.310 INFO:tasks.workunit:cleanup=True 2024-09-17T15:52:56.311 DEBUG:teuthology.orchestra.run.smithi162:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-17T15:52:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:52:56.349 INFO:teuthology.orchestra.run.smithi162.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-17T15:52:56.349 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-17T15:52:56.404 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-17T15:52:56.404 DEBUG:teuthology.orchestra.run.smithi162:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-17T15:52:56.458 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:52:56.514 INFO:tasks.workunit.client.0.smithi162.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-17T15:53:46.522 INFO:tasks.workunit.client.0.smithi162.stderr:Note: switching to '1889a165b5128787ec693450a3d1c5df5d6dc203'. 2024-09-17T15:53:46.522 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-09-17T15:53:46.522 INFO:tasks.workunit.client.0.smithi162.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-17T15:53:46.522 INFO:tasks.workunit.client.0.smithi162.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-17T15:53:46.522 INFO:tasks.workunit.client.0.smithi162.stderr:state without impacting any branches by switching back to a branch. 2024-09-17T15:53:46.523 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-09-17T15:53:46.523 INFO:tasks.workunit.client.0.smithi162.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-17T15:53:46.523 INFO:tasks.workunit.client.0.smithi162.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-17T15:53:46.523 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-09-17T15:53:46.523 INFO:tasks.workunit.client.0.smithi162.stderr: git switch -c 2024-09-17T15:53:46.523 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-09-17T15:53:46.523 INFO:tasks.workunit.client.0.smithi162.stderr:Or undo this operation with: 2024-09-17T15:53:46.523 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-09-17T15:53:46.523 INFO:tasks.workunit.client.0.smithi162.stderr: git switch - 2024-09-17T15:53:46.523 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-09-17T15:53:46.523 INFO:tasks.workunit.client.0.smithi162.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-17T15:53:46.523 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-09-17T15:53:46.523 INFO:tasks.workunit.client.0.smithi162.stderr:HEAD is now at 1889a165b51 Merge branch 'wip-matanb-cleanup-snapmap' of https://github.com/Matan-B/ceph into wip-bharath10-testing-2024-09-17-1348 2024-09-17T15:53:46.526 DEBUG:teuthology.orchestra.run.smithi162:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-17T15:53:46.582 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-17T15:53:46.582 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-17T15:53:46.637 INFO:tasks.workunit:Running workunits matching erasure-code on client.0... 2024-09-17T15:53:46.638 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code-plugins.sh... 2024-09-17T15:53:46.638 DEBUG:teuthology.orchestra.run.smithi162:workunit test erasure-code/test-erasure-code-plugins.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh 2024-09-17T15:53:46.707 INFO:tasks.workunit.client.0.smithi162.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-09-17T15:53:46.707 INFO:tasks.workunit.client.0.smithi162.stderr:++ TIMEOUT=300 2024-09-17T15:53:46.707 INFO:tasks.workunit.client.0.smithi162.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-17T15:53:46.707 INFO:tasks.workunit.client.0.smithi162.stderr:++ MAX_TIMEOUT=15 2024-09-17T15:53:46.707 INFO:tasks.workunit.client.0.smithi162.stderr:++ PG_NUM=4 2024-09-17T15:53:46.707 INFO:tasks.workunit.client.0.smithi162.stderr:++ TMPDIR=/tmp 2024-09-17T15:53:46.707 INFO:tasks.workunit.client.0.smithi162.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-17T15:53:46.707 INFO:tasks.workunit.client.0.smithi162.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-17T15:53:46.707 INFO:tasks.workunit.client.0.smithi162.stderr:+++ uname 2024-09-17T15:53:46.708 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' Linux = FreeBSD ']' 2024-09-17T15:53:46.708 INFO:tasks.workunit.client.0.smithi162.stderr:++ SED=sed 2024-09-17T15:53:46.708 INFO:tasks.workunit.client.0.smithi162.stderr:++ AWK=awk 2024-09-17T15:53:46.709 INFO:tasks.workunit.client.0.smithi162.stderr:+++ stty -a 2024-09-17T15:53:46.709 INFO:tasks.workunit.client.0.smithi162.stderr:+++ head -1 2024-09-17T15:53:46.709 INFO:tasks.workunit.client.0.smithi162.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-17T15:53:46.709 INFO:tasks.workunit.client.0.smithi162.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-17T15:53:46.709 INFO:tasks.workunit.client.0.smithi162.stderr:++ termwidth= 2024-09-17T15:53:46.709 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-17T15:53:46.709 INFO:tasks.workunit.client.0.smithi162.stderr:++ DIFFCOLOPTS='-y ' 2024-09-17T15:53:46.709 INFO:tasks.workunit.client.0.smithi162.stderr:++ KERNCORE=kernel.core_pattern 2024-09-17T15:53:46.710 INFO:tasks.workunit.client.0.smithi162.stderr:++ EXTRA_OPTS= 2024-09-17T15:53:46.712 INFO:tasks.workunit.client.0.smithi162.stderr:++ test '' = TESTS 2024-09-17T15:53:46.712 INFO:tasks.workunit.client.0.smithi162.stderr:++ uname -m 2024-09-17T15:53:46.715 INFO:tasks.workunit.client.0.smithi162.stderr:+ arch=x86_64 2024-09-17T15:53:46.715 INFO:tasks.workunit.client.0.smithi162.stderr:+ case $arch in 2024-09-17T15:53:46.715 INFO:tasks.workunit.client.0.smithi162.stderr:+ legacy_jerasure_plugins=(jerasure_generic jerasure_sse3 jerasure_sse4) 2024-09-17T15:53:46.715 INFO:tasks.workunit.client.0.smithi162.stderr:+ legacy_shec_plugins=(shec_generic shec_sse3 shec_sse4) 2024-09-17T15:53:46.715 INFO:tasks.workunit.client.0.smithi162.stderr:+ plugins=(jerasure shec lrc isa) 2024-09-17T15:53:46.715 INFO:tasks.workunit.client.0.smithi162.stderr:+ main test-erasure-code-plugins 2024-09-17T15:53:46.715 INFO:tasks.workunit.client.0.smithi162.stderr:+ local dir=td/test-erasure-code-plugins 2024-09-17T15:53:46.715 INFO:tasks.workunit.client.0.smithi162.stderr:+ shift 2024-09-17T15:53:46.715 INFO:tasks.workunit.client.0.smithi162.stderr:+ shopt -s -o xtrace 2024-09-17T15:53:46.715 INFO:tasks.workunit.client.0.smithi162.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-17T15:53:46.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T15:53:46.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T15:53:46.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-17T15:53:46.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-17T15:53:46.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-17T15:53:46.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-17T15:53:46.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-17T15:53:46.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-17T15:53:46.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/test-erasure-code-plugins 2024-09-17T15:53:46.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:26: run: local dir=td/test-erasure-code-plugins 2024-09-17T15:53:46.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:27: run: shift 2024-09-17T15:53:46.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: export CEPH_MON=127.0.0.1:17110 2024-09-17T15:53:46.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: CEPH_MON=127.0.0.1:17110 2024-09-17T15:53:46.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:30: run: export CEPH_ARGS 2024-09-17T15:53:46.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: uuidgen 2024-09-17T15:53:46.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: CEPH_ARGS+='--fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none ' 2024-09-17T15:53:46.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:17110 ' 2024-09-17T15:53:46.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: set 2024-09-17T15:53:46.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-17T15:53:46.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: local 'funcs=TEST_ec_profile_warning 2024-09-17T15:53:46.720 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_preload_no_warning 2024-09-17T15:53:46.720 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_preload_no_warning_default 2024-09-17T15:53:46.720 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_preload_warning' 2024-09-17T15:53:46.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-17T15:53:46.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_warning td/test-erasure-code-plugins 2024-09-17T15:53:46.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:92: TEST_ec_profile_warning: local dir=td/test-erasure-code-plugins 2024-09-17T15:53:46.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:94: TEST_ec_profile_warning: setup td/test-erasure-code-plugins 2024-09-17T15:53:46.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-17T15:53:46.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-17T15:53:46.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:53:46.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:53:46.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:53:46.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:53:46.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:53:46.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:53:46.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:53:46.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:53:46.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:53:46.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:53:46.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:53:46.723 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:53:46.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:53:46.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:53:46.723 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:53:46.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:53:46.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:53:46.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:53:46.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:53:46.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:53:46.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:53:46.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:53:46.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:53:46.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:53:46.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:53:46.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:53:46.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:53:46.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:53:46.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-17T15:53:46.730 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T15:53:46.730 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:53:46.730 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:53:46.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27518 2024-09-17T15:53:46.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T15:53:46.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-17T15:53:46.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-17T15:53:46.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T15:53:46.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-17T15:53:46.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:95: TEST_ec_profile_warning: run_mon td/test-erasure-code-plugins a 2024-09-17T15:53:46.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-17T15:53:46.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T15:53:46.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T15:53:46.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T15:53:46.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-17T15:53:46.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-09-17T15:53:47.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T15:53:47.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:53:47.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:53:47.283 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:53:47.284 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:53:47.284 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:53:47.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:53:47.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T15:53:47.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T15:53:47.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T15:53:47.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:53:47.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:53:47.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T15:53:47.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T15:53:47.461 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:53:47.462 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:53:47.462 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:53:47.462 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:53:47.462 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:53:47.462 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:53:47.462 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:53:47.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:53:47.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get fsid 2024-09-17T15:53:47.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T15:53:47.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:53:47.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:53:47.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T15:53:47.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T15:53:47.545 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:53:47.545 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:53:47.545 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:53:47.545 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:53:47.545 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:53:47.545 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:53:47.545 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:53:47.546 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:53:47.546 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get mon_host 2024-09-17T15:53:47.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:96: TEST_ec_profile_warning: run_mgr td/test-erasure-code-plugins x 2024-09-17T15:53:47.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-17T15:53:47.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T15:53:47.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T15:53:47.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T15:53:47.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-17T15:53:47.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T15:53:48.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T15:53:48.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:53:48.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:53:48.467 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:53:48.467 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:53:48.467 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:53:48.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:53:48.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:53:48.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:53:48.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: seq 0 2 2024-09-17T15:53:48.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-17T15:53:48.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 0 2024-09-17T15:53:48.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-17T15:53:48.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T15:53:48.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T15:53:48.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T15:53:48.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-17T15:53:48.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-17T15:53:48.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T15:53:48.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T15:53:48.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T15:53:48.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-17T15:53:48.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-17T15:53:48.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T15:53:48.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T15:53:48.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-17T15:53:48.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T15:53:48.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:53:48.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:53:48.502 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:53:48.502 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:53:48.502 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:53:48.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:53:48.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:53:48.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T15:53:48.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T15:53:48.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T15:53:48.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-17T15:53:48.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-17T15:53:48.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T15:53:48.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T15:53:48.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T15:53:48.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T15:53:48.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T15:53:48.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T15:53:48.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-17T15:53:48.503 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T15:53:48.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=01a10122-e3a9-47a0-aeaa-4c974605dbc8 2024-09-17T15:53:48.504 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 01a10122-e3a9-47a0-aeaa-4c974605dbc8 2024-09-17T15:53:48.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 01a10122-e3a9-47a0-aeaa-4c974605dbc8' 2024-09-17T15:53:48.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T15:53:48.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAMpulmUt/DHhAAUAQjCCb+r+4tCovfTY1n3g== 2024-09-17T15:53:48.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAMpulmUt/DHhAAUAQjCCb+r+4tCovfTY1n3g=="}' 2024-09-17T15:53:48.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 01a10122-e3a9-47a0-aeaa-4c974605dbc8 -i td/test-erasure-code-plugins/0/new.json 2024-09-17T15:53:48.787 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:53:48.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-17T15:53:48.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMpulmUt/DHhAAUAQjCCb+r+4tCovfTY1n3g== --osd-uuid 01a10122-e3a9-47a0-aeaa-4c974605dbc8 2024-09-17T15:53:48.859 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:53:48.857+0000 7fc215745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:53:48.867 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:53:48.863+0000 7fc215745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:53:48.867 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:53:48.864+0000 7fc215745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:53:48.868 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:53:48.864+0000 7fc215745640 -1 bdev(0x563625c55c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-17T15:53:48.868 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:53:48.864+0000 7fc215745640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-17T15:53:54.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-17T15:53:54.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T15:53:54.614 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T15:53:54.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T15:53:54.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T15:53:55.134 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T15:53:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T15:53:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T15:53:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T15:53:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T15:53:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T15:53:55.161 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:53:55.159+0000 7f5c13a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:53:55.192 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:53:55.190+0000 7f5c13a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:53:55.193 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:53:55.192+0000 7f5c13a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:53:55.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T15:53:55.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T15:53:55.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T15:53:55.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T15:53:55.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T15:53:55.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:53:55.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T15:53:55.557 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:53:55.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:53:55.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:53:55.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:53:56.283 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:53:56.282+0000 7f5c13a0d640 -1 Falling back to public interface 2024-09-17T15:53:56.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:53:56.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:53:56.975 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T15:53:56.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T15:53:56.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:53:56.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:53:57.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:53:58.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:53:58.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:53:58.393 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T15:53:58.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T15:53:58.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:53:58.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:53:58.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:53:59.070 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:53:59.068+0000 7f5c13a0d640 -1 osd.0 0 log_to_monitors true 2024-09-17T15:53:59.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:53:59.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:53:59.793 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T15:53:59.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T15:53:59.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:53:59.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:00.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:01.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:01.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:01.201 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T15:54:01.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T15:54:01.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:01.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:01.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:02.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:02.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:02.599 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T15:54:02.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T15:54:02.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:02.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:02.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:03.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:03.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:03.990 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T15:54:03.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T15:54:03.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:03.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:04.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:05.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:05.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:05.401 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T15:54:05.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T15:54:05.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:05.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:05.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:06.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:06.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:06.879 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T15:54:06.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T15:54:06.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:06.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:07.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:08.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:08.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:08.275 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T15:54:08.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T15:54:08.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:08.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:08.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:09.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:09.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:09.703 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T15:54:09.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T15:54:09.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:09.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:10.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:11.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:11.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:11.160 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T15:54:11.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T15:54:11.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:11.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:11.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:12.651 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T15:54:12.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T15:54:12.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:12.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:13.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:14.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:14.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:14.185 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T15:54:14.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T15:54:14.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:14.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:15.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:16.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:16.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:16.153 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T15:54:16.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T15:54:16.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:16.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:16.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:17.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:17.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:17.710 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T15:54:17.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T15:54:17.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:17.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:18.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:19.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:19.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:19.417 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T15:54:19.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T15:54:19.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:19.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:20.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:21.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:21.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:21.670 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T15:54:21.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T15:54:21.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:21.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:22.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:23.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:23.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:23.309 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T15:54:23.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T15:54:23.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:23.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:23.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:24.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:24.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:24.927 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-09-17T15:54:24.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T15:54:24.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:24.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:54:25.354 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3041223446,v1:127.0.0.1:6803/3041223446] [v2:127.0.0.1:6804/3041223446,v1:127.0.0.1:6805/3041223446] exists,up 01a10122-e3a9-47a0-aeaa-4c974605dbc8 2024-09-17T15:54:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T15:54:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T15:54:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T15:54:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-17T15:54:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 1 2024-09-17T15:54:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-17T15:54:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T15:54:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T15:54:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T15:54:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/1 2024-09-17T15:54:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-17T15:54:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T15:54:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T15:54:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T15:54:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2024-09-17T15:54:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2024-09-17T15:54:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T15:54:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T15:54:25.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-17T15:54:25.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T15:54:25.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:54:25.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:54:25.360 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:54:25.360 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:54:25.360 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:54:25.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:54:25.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:54:25.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T15:54:25.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T15:54:25.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T15:54:25.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-17T15:54:25.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-17T15:54:25.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T15:54:25.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T15:54:25.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T15:54:25.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T15:54:25.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T15:54:25.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T15:54:25.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/1 2024-09-17T15:54:25.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T15:54:25.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=755850c3-ec2e-401e-8a3f-ae5e0d290fc5 2024-09-17T15:54:25.361 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 755850c3-ec2e-401e-8a3f-ae5e0d290fc5 2024-09-17T15:54:25.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 755850c3-ec2e-401e-8a3f-ae5e0d290fc5' 2024-09-17T15:54:25.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T15:54:25.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAxpulmVrA9FhAAx7YZqgTEsszBAcyBtE76Ug== 2024-09-17T15:54:25.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAxpulmVrA9FhAAx7YZqgTEsszBAcyBtE76Ug=="}' 2024-09-17T15:54:25.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 755850c3-ec2e-401e-8a3f-ae5e0d290fc5 -i td/test-erasure-code-plugins/1/new.json 2024-09-17T15:54:25.862 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T15:54:25.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/1/new.json 2024-09-17T15:54:25.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxpulmVrA9FhAAx7YZqgTEsszBAcyBtE76Ug== --osd-uuid 755850c3-ec2e-401e-8a3f-ae5e0d290fc5 2024-09-17T15:54:25.901 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:54:25.899+0000 7f7d5b4e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:54:25.904 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:54:25.901+0000 7f7d5b4e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:54:25.904 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:54:25.902+0000 7f7d5b4e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:54:25.905 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:54:25.902+0000 7f7d5b4e8640 -1 bdev(0x55f99b3efc00 td/test-erasure-code-plugins/1/block) open stat got: (1) Operation not permitted 2024-09-17T15:54:25.905 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:54:25.902+0000 7f7d5b4e8640 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2024-09-17T15:54:31.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2024-09-17T15:54:31.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T15:54:31.650 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T15:54:31.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T15:54:31.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T15:54:32.187 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T15:54:32.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T15:54:32.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T15:54:32.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T15:54:32.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T15:54:32.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T15:54:32.212 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:54:32.211+0000 7feb23810640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:54:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:54:32.230+0000 7feb23810640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:54:32.233 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:54:32.231+0000 7feb23810640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:54:32.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T15:54:32.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T15:54:32.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T15:54:32.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T15:54:32.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T15:54:32.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:32.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T15:54:32.630 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:54:32.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:32.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:33.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:33.324 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:54:33.320+0000 7feb23810640 -1 Falling back to public interface 2024-09-17T15:54:34.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:34.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:34.048 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T15:54:34.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T15:54:34.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:34.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:34.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:35.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:35.441 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T15:54:35.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:35.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T15:54:35.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:35.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:35.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:36.519 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:54:36.517+0000 7feb23810640 -1 osd.1 0 log_to_monitors true 2024-09-17T15:54:36.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:36.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:36.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T15:54:36.860 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T15:54:36.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:36.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:37.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:38.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:38.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:38.348 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T15:54:38.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T15:54:38.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:38.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:38.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:39.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:39.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:39.832 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T15:54:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T15:54:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:39.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:40.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:41.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:41.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:41.429 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T15:54:41.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T15:54:41.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:41.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:41.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:42.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:42.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:42.819 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T15:54:42.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T15:54:42.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:42.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:43.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:44.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:44.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:44.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T15:54:44.334 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T15:54:44.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:44.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:44.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:45.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:45.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:45.831 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T15:54:45.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T15:54:45.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:45.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:46.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:47.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:47.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:47.420 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T15:54:47.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T15:54:47.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:47.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:47.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:48.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:48.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:48.813 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T15:54:48.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T15:54:48.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:48.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:49.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:50.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:50.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:50.730 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T15:54:50.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T15:54:50.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:50.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:51.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:52.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:52.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:52.759 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T15:54:52.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T15:54:52.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:52.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:53.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:54.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:54.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:54.802 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T15:54:54.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T15:54:54.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:54.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:55.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:56.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:56.983 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T15:54:56.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T15:54:56.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:56.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:57.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:54:58.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:54:58.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:54:58.744 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T15:54:58.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T15:54:58.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:54:58.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T15:54:59.180 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/637974431,v1:127.0.0.1:6811/637974431] [v2:127.0.0.1:6812/637974431,v1:127.0.0.1:6813/637974431] exists,up 755850c3-ec2e-401e-8a3f-ae5e0d290fc5 2024-09-17T15:54:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T15:54:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T15:54:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T15:54:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-17T15:54:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 2 2024-09-17T15:54:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-17T15:54:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T15:54:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T15:54:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T15:54:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/2 2024-09-17T15:54:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-17T15:54:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T15:54:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T15:54:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T15:54:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2024-09-17T15:54:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2024-09-17T15:54:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T15:54:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T15:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-17T15:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T15:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T15:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T15:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T15:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-17T15:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-17T15:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T15:54:59.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T15:54:59.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T15:54:59.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T15:54:59.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T15:54:59.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T15:54:59.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/2 2024-09-17T15:54:59.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T15:54:59.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=612014d6-910d-40d6-a887-6a2a887f9858 2024-09-17T15:54:59.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 612014d6-910d-40d6-a887-6a2a887f9858' 2024-09-17T15:54:59.190 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 612014d6-910d-40d6-a887-6a2a887f9858 2024-09-17T15:54:59.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T15:54:59.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBTpulmKo3aCxAABHaphWlcKwzzljKXEefC1g== 2024-09-17T15:54:59.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBTpulmKo3aCxAABHaphWlcKwzzljKXEefC1g=="}' 2024-09-17T15:54:59.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 612014d6-910d-40d6-a887-6a2a887f9858 -i td/test-erasure-code-plugins/2/new.json 2024-09-17T15:54:59.720 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T15:54:59.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/2/new.json 2024-09-17T15:54:59.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTpulmKo3aCxAABHaphWlcKwzzljKXEefC1g== --osd-uuid 612014d6-910d-40d6-a887-6a2a887f9858 2024-09-17T15:54:59.759 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:54:59.757+0000 7f2160fa2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:54:59.773 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:54:59.771+0000 7f2160fa2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:54:59.774 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:54:59.772+0000 7f2160fa2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:54:59.775 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:54:59.773+0000 7f2160fa2640 -1 bdev(0x55f1737ffc00 td/test-erasure-code-plugins/2/block) open stat got: (1) Operation not permitted 2024-09-17T15:54:59.775 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:54:59.773+0000 7f2160fa2640 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2024-09-17T15:55:05.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2024-09-17T15:55:05.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T15:55:05.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T15:55:05.562 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T15:55:05.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T15:55:06.100 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T15:55:06.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T15:55:06.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T15:55:06.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T15:55:06.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T15:55:06.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T15:55:06.125 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:55:06.122+0000 7f55ea016640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:55:06.151 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:55:06.149+0000 7f55ea016640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:55:06.152 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:55:06.150+0000 7f55ea016640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:55:06.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T15:55:06.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T15:55:06.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T15:55:06.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T15:55:06.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T15:55:06.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:06.592 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:55:06.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T15:55:06.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:06.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:07.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:07.506 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:55:07.504+0000 7f55ea016640 -1 Falling back to public interface 2024-09-17T15:55:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T15:55:08.009 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T15:55:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:08.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:09.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:09.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:09.444 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T15:55:09.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T15:55:09.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:09.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:09.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:09.995 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:55:09.993+0000 7f55ea016640 -1 osd.2 0 log_to_monitors true 2024-09-17T15:55:10.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:10.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:10.838 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T15:55:10.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T15:55:10.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:10.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:11.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:12.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:12.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:12.251 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T15:55:12.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T15:55:12.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:12.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:12.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:13.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:13.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:13.644 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T15:55:13.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T15:55:13.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:13.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:14.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:15.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:15.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:15.043 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T15:55:15.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T15:55:15.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:15.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:15.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:16.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:16.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:16.435 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T15:55:16.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T15:55:16.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:16.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:16.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:17.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:17.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:17.932 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T15:55:17.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T15:55:17.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:17.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:18.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:19.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:19.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:19.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T15:55:19.379 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T15:55:19.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:19.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:19.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:20.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:20.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:20.887 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T15:55:20.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T15:55:20.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:20.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:21.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:22.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:22.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:22.282 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T15:55:22.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T15:55:22.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:22.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:22.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:23.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:23.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:23.678 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T15:55:23.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T15:55:23.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:23.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:24.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:25.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:25.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:25.155 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T15:55:25.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T15:55:25.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:25.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:25.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:26.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:26.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:26.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T15:55:26.919 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T15:55:26.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:26.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:27.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:28.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:28.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:28.787 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T15:55:28.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T15:55:28.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:28.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:29.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:30.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:30.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:30.582 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T15:55:30.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T15:55:30.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:30.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:31.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:32.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:32.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:32.359 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T15:55:32.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T15:55:32.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:32.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:32.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:33.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:33.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:33.756 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T15:55:33.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T15:55:33.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:33.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T15:55:34.153 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2061032698,v1:127.0.0.1:6819/2061032698] [v2:127.0.0.1:6820/2061032698,v1:127.0.0.1:6821/2061032698] exists,up 612014d6-910d-40d6-a887-6a2a887f9858 2024-09-17T15:55:34.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T15:55:34.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T15:55:34.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T15:55:34.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:100: TEST_ec_profile_warning: create_rbd_pool 2024-09-17T15:55:34.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-17T15:55:34.579 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-09-17T15:55:34.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-17T15:55:34.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T15:55:35.078 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T15:55:35.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T15:55:36.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-17T15:55:38.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:101: TEST_ec_profile_warning: wait_for_clean 2024-09-17T15:55:38.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T15:55:38.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T15:55:38.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T15:55:38.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T15:55:38.191 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T15:55:38.191 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T15:55:38.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T15:55:38.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T15:55:38.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T15:55:38.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T15:55:38.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T15:55:38.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T15:55:38.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T15:55:38.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T15:55:38.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T15:55:38.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T15:55:38.712 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T15:55:38.712 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T15:55:38.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T15:55:38.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T15:55:38.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T15:55:38.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-17T15:55:38.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-17T15:55:38.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-17T15:55:38.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T15:55:38.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T15:55:38.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-17T15:55:38.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-17T15:55:38.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672974' 2024-09-17T15:55:38.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T15:55:38.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T15:55:39.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509445 2024-09-17T15:55:39.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509445 2024-09-17T15:55:39.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672974 2-64424509445' 2024-09-17T15:55:39.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T15:55:39.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-17T15:55:39.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T15:55:39.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T15:55:39.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-17T15:55:39.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T15:55:39.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-17T15:55:39.070 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836503 2024-09-17T15:55:39.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-17T15:55:39.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T15:55:39.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836503 2024-09-17T15:55:39.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T15:55:40.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T15:55:40.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T15:55:40.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836503 2024-09-17T15:55:40.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T15:55:40.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-17T15:55:40.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T15:55:40.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T15:55:40.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-17T15:55:40.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T15:55:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-17T15:55:40.934 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672974 2024-09-17T15:55:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-17T15:55:40.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T15:55:41.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672974 2024-09-17T15:55:41.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T15:55:41.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509445 2024-09-17T15:55:41.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T15:55:41.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T15:55:41.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509445 2024-09-17T15:55:41.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T15:55:41.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509445 2024-09-17T15:55:41.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-17T15:55:41.353 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509445 2024-09-17T15:55:41.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T15:55:41.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509445 -lt 64424509445 2024-09-17T15:55:41.767 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T15:55:41.767 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T15:55:41.767 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T15:55:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T15:55:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T15:55:42.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T15:55:42.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T15:55:42.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T15:55:42.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T15:55:42.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T15:55:42.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T15:55:42.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T15:55:42.704 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T15:55:42.704 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T15:55:42.704 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T15:55:43.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T15:55:43.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T15:55:43.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T15:55:43.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-17T15:55:43.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_generic crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_generic 2024-09-17T15:55:43.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-17T15:55:43.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:55:43.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:55:43.864 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:55:43.864 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:55:43.864 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:55:43.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:55:43.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-17T15:55:43.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T15:55:43.933 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:55:43.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-17T15:55:43.945 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:43.536+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-17T15:55:43.945 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:43.849+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-17T15:55:43.945 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:43.849+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-17T15:55:43.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-17T15:55:43.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse3 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse3 2024-09-17T15:55:44.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-17T15:55:44.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:55:44.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:55:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:55:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:55:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:55:44.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:55:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-17T15:55:44.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T15:55:44.724 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:55:44.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-17T15:55:44.736 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:44.309+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-17T15:55:44.736 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:44.638+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-17T15:55:44.736 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:44.638+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-17T15:55:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-17T15:55:44.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse4 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse4 2024-09-17T15:55:45.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-17T15:55:45.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:55:45.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:55:45.267 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:55:45.268 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:55:45.268 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:55:45.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:55:45.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-17T15:55:45.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T15:55:45.338 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:55:45.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-17T15:55:45.350 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:45.008+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-17T15:55:45.351 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:45.253+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-17T15:55:45.351 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:45.253+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-17T15:55:45.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-17T15:55:45.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_generic crush-failure-domain=osd plugin=shec_generic 2024-09-17T15:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-17T15:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-17T15:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T15:55:46.010 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:55:46.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-17T15:55:46.023 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:45.656+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-17T15:55:46.023 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:45.925+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-17T15:55:46.023 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:45.925+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-17T15:55:46.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-17T15:55:46.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse3 crush-failure-domain=osd plugin=shec_sse3 2024-09-17T15:55:46.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-17T15:55:46.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:55:46.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:55:46.624 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:55:46.624 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:55:46.625 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:55:46.625 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:55:46.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-17T15:55:46.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T15:55:46.694 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:55:46.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-17T15:55:46.707 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:46.297+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-17T15:55:46.708 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:46.609+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-17T15:55:46.708 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:46.609+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-17T15:55:46.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-17T15:55:46.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse4 crush-failure-domain=osd plugin=shec_sse4 2024-09-17T15:55:47.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-17T15:55:47.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:55:47.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:55:47.260 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:55:47.260 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:55:47.260 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:55:47.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:55:47.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-17T15:55:47.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T15:55:47.331 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:55:47.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-17T15:55:47.343 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:47.020+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-17T15:55:47.344 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:47.244+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e31 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-17T15:55:47.344 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:55:47.244+0000 7f3e9bbd4640 0 mon.a@0(leader).osd e31 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-17T15:55:47.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:115: TEST_ec_profile_warning: teardown td/test-erasure-code-plugins 2024-09-17T15:55:47.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:55:47.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:55:47.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:55:47.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:55:47.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:55:47.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:55:47.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:55:47.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:55:47.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:55:47.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:55:47.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:55:47.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:55:47.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:55:47.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:55:47.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:55:47.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:55:47.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:55:47.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:55:47.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:55:47.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:55:47.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:55:47.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:55:47.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:55:47.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:55:47.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:55:47.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:55:47.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:55:47.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:55:47.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-17T15:55:47.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning td/test-erasure-code-plugins 2024-09-17T15:55:47.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:58: TEST_preload_no_warning: local dir=td/test-erasure-code-plugins 2024-09-17T15:55:47.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-17T15:55:47.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-17T15:55:47.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-17T15:55:47.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-17T15:55:47.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:55:47.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:55:47.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:55:47.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:55:47.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:55:47.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:55:47.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:55:47.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:55:47.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:55:47.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:55:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:55:47.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:55:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:55:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:55:47.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:55:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:55:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:55:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:55:47.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:55:47.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:55:47.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:55:47.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:55:47.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:55:47.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:55:47.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:55:47.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:55:47.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:55:47.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:55:47.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-17T15:55:47.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T15:55:47.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:55:47.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:55:47.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27518 2024-09-17T15:55:47.485 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T15:55:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T15:55:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T15:55:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-17T15:55:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure 2024-09-17T15:55:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-17T15:55:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T15:55:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T15:55:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T15:55:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-17T15:55:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure 2024-09-17T15:55:47.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T15:55:47.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:55:47.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:55:47.939 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:55:47.939 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:55:47.939 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:55:47.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:55:47.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure 2024-09-17T15:55:48.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T15:55:48.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T15:55:48.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:55:48.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:55:48.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T15:55:48.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T15:55:48.116 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:55:48.116 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:55:48.117 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:55:48.117 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:55:48.117 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:55:48.117 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:55:48.117 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:55:48.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:55:48.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get fsid 2024-09-17T15:55:48.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T15:55:48.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:55:48.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:55:48.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T15:55:48.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T15:55:48.195 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:55:48.195 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:55:48.195 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:55:48.195 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:55:48.195 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:55:48.195 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:55:48.195 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:55:48.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:55:48.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get mon_host 2024-09-17T15:55:48.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-17T15:55:48.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-17T15:55:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T15:55:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T15:55:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T15:55:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-17T15:55:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T15:55:49.110 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T15:55:49.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:55:49.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:55:49.111 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:55:49.111 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:55:49.111 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:55:49.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:55:49.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:55:49.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:55:49.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-17T15:55:49.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:55:49.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:55:49.140 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:55:49.140 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:55:49.140 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:55:49.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:55:49.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-17T15:55:49.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T15:55:49.210 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:55:49.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure 2024-09-17T15:55:49.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-17T15:55:49.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T15:55:49.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T15:55:49.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T15:55:49.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-17T15:55:49.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-17T15:55:49.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T15:55:49.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T15:55:49.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T15:55:49.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-17T15:55:49.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-17T15:55:49.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T15:55:49.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T15:55:49.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-17T15:55:49.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T15:55:49.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:55:49.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:55:49.222 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:55:49.222 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:55:49.223 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:55:49.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:55:49.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:55:49.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T15:55:49.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T15:55:49.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T15:55:49.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-17T15:55:49.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-17T15:55:49.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T15:55:49.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T15:55:49.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T15:55:49.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T15:55:49.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T15:55:49.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure 2024-09-17T15:55:49.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-17T15:55:49.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T15:55:49.225 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 99c2bd0d-263b-40a7-becd-2434cbbdc064 2024-09-17T15:55:49.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=99c2bd0d-263b-40a7-becd-2434cbbdc064 2024-09-17T15:55:49.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 99c2bd0d-263b-40a7-becd-2434cbbdc064' 2024-09-17T15:55:49.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T15:55:49.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCFpulmWoIZDhAA2xVnoZsbPyrxYU/RkpdbXw== 2024-09-17T15:55:49.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCFpulmWoIZDhAA2xVnoZsbPyrxYU/RkpdbXw=="}' 2024-09-17T15:55:49.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 99c2bd0d-263b-40a7-becd-2434cbbdc064 -i td/test-erasure-code-plugins/0/new.json 2024-09-17T15:55:49.488 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:55:49.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-17T15:55:49.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure --mkfs --key AQCFpulmWoIZDhAA2xVnoZsbPyrxYU/RkpdbXw== --osd-uuid 99c2bd0d-263b-40a7-becd-2434cbbdc064 2024-09-17T15:55:49.527 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:55:49.525+0000 7fbe92a62640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:55:49.558 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:55:49.556+0000 7fbe92a62640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:55:49.558 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:55:49.557+0000 7fbe92a62640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:55:49.559 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:55:49.557+0000 7fbe92a62640 -1 bdev(0x55bc74781c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-17T15:55:49.559 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:55:49.557+0000 7fbe92a62640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-17T15:55:55.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-17T15:55:55.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T15:55:55.275 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T15:55:55.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T15:55:55.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T15:55:55.806 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T15:55:55.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T15:55:55.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure 2024-09-17T15:55:55.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T15:55:55.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T15:55:55.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T15:55:55.832 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:55:55.829+0000 7fe74a21c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:55:55.845 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:55:55.842+0000 7fe74a21c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:55:55.846 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:55:55.843+0000 7fe74a21c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:55:56.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T15:55:56.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T15:55:56.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T15:55:56.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T15:55:56.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T15:55:56.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:56.258 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:55:56.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T15:55:56.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:56.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:55:56.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:56.954 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:55:56.952+0000 7fe74a21c640 -1 Falling back to public interface 2024-09-17T15:55:57.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:57.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:57.685 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T15:55:57.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T15:55:57.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:57.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:55:58.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:55:59.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:55:59.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:55:59.111 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T15:55:59.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T15:55:59.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:55:59.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:55:59.379 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:55:59.376+0000 7fe74a21c640 -1 osd.0 0 log_to_monitors true 2024-09-17T15:55:59.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:00.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:00.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:00.556 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T15:56:00.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T15:56:00.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:00.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:00.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:01.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:01.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:01.949 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T15:56:01.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T15:56:01.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:01.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:02.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:03.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:03.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:03.469 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T15:56:03.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T15:56:03.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:03.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:03.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:04.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:04.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:04.870 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T15:56:04.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T15:56:04.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:04.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:05.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:06.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:06.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:06.281 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T15:56:06.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T15:56:06.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:06.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:06.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:07.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:07.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:07.699 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T15:56:07.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T15:56:07.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:07.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:08.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:09.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:09.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:09.229 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T15:56:09.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T15:56:09.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:09.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:09.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:10.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:10.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:10.622 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T15:56:10.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T15:56:10.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:10.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:11.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:12.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:12.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:12.032 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T15:56:12.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T15:56:12.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:12.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:12.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:13.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:13.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:13.487 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T15:56:13.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T15:56:13.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:13.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:14.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:15.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:15.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:15.276 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T15:56:15.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T15:56:15.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:15.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:15.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:16.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:16.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:16.803 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T15:56:16.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T15:56:16.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:16.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:17.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:18.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:18.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:18.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T15:56:18.413 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T15:56:18.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:18.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:19.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:20.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:20.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:20.349 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T15:56:20.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T15:56:20.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:20.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:21.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:22.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:22.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:22.127 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T15:56:22.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T15:56:22.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:22.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:22.518 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2307726500,v1:127.0.0.1:6803/2307726500] [v2:127.0.0.1:6804/2307726500,v1:127.0.0.1:6805/2307726500] exists,up 99c2bd0d-263b-40a7-becd-2434cbbdc064 2024-09-17T15:56:22.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T15:56:22.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T15:56:22.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T15:56:22.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-17T15:56:22.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T15:56:22.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T15:56:22.520 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:56:22.521 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:22.521 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:22.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-osd.0.asok 2024-09-17T15:56:22.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-17T15:56:22.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-osd.0.asok log flush 2024-09-17T15:56:22.598 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:56:22.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-17T15:56:22.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-17T15:56:22.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-17T15:56:22.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:56:22.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:56:22.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:56:22.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:56:22.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:56:22.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:56:22.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:56:22.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:56:22.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:56:22.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:56:22.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:56:22.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:56:22.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:56:22.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:56:22.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:56:22.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:56:22.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:56:22.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:56:22.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:56:22.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:56:22.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:56:22.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:56:22.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:56:22.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:22.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:22.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:56:22.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:56:22.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:56:22.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-17T15:56:22.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-17T15:56:22.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-17T15:56:22.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-17T15:56:22.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:56:22.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:56:22.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:56:22.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:56:22.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:56:22.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:56:22.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:56:22.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:56:22.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:56:22.730 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:56:22.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:56:22.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:56:22.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:56:22.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:56:22.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:56:22.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:56:22.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:56:22.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:56:22.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:56:22.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:56:22.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:56:22.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:56:22.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:56:22.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:22.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:22.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:56:22.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:56:22.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:56:22.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-17T15:56:22.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T15:56:22.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:22.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:22.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27518 2024-09-17T15:56:22.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T15:56:22.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T15:56:22.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T15:56:22.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-17T15:56:22.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec 2024-09-17T15:56:22.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-17T15:56:22.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T15:56:22.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T15:56:22.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T15:56:22.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-17T15:56:22.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec 2024-09-17T15:56:23.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T15:56:23.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:56:23.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:56:23.194 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:56:23.194 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:23.194 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:23.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:56:23.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec 2024-09-17T15:56:23.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T15:56:23.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T15:56:23.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:56:23.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:56:23.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T15:56:23.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T15:56:23.379 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:56:23.379 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:56:23.379 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:56:23.379 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:56:23.379 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:23.379 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:23.379 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:56:23.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:56:23.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get fsid 2024-09-17T15:56:23.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T15:56:23.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:56:23.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:56:23.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T15:56:23.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T15:56:23.460 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:56:23.460 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:56:23.460 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:56:23.460 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:56:23.460 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:23.460 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:23.460 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:56:23.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:56:23.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get mon_host 2024-09-17T15:56:23.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-17T15:56:23.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-17T15:56:23.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T15:56:23.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T15:56:23.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T15:56:23.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-17T15:56:23.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T15:56:24.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T15:56:24.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:56:24.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:56:24.391 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:56:24.391 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:24.391 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:24.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:56:24.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:56:24.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:56:24.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-17T15:56:24.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:56:24.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:56:24.419 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:56:24.419 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:24.419 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:24.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:56:24.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-17T15:56:24.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T15:56:24.508 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:56:24.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec 2024-09-17T15:56:24.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-17T15:56:24.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T15:56:24.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T15:56:24.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T15:56:24.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-17T15:56:24.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-17T15:56:24.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T15:56:24.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T15:56:24.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T15:56:24.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-17T15:56:24.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-17T15:56:24.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T15:56:24.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T15:56:24.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-17T15:56:24.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T15:56:24.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:56:24.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:56:24.520 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:56:24.520 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:24.520 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:24.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:56:24.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:56:24.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T15:56:24.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T15:56:24.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T15:56:24.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-17T15:56:24.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-17T15:56:24.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T15:56:24.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T15:56:24.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T15:56:24.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T15:56:24.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T15:56:24.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec 2024-09-17T15:56:24.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-17T15:56:24.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T15:56:24.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=01b3e74d-0aa3-49f1-8d4c-a82dbb47aec2 2024-09-17T15:56:24.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 01b3e74d-0aa3-49f1-8d4c-a82dbb47aec2' 2024-09-17T15:56:24.522 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 01b3e74d-0aa3-49f1-8d4c-a82dbb47aec2 2024-09-17T15:56:24.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T15:56:24.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCopulmJNTeHxAAKjcaLvHE3gHKardR8nTuiw== 2024-09-17T15:56:24.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCopulmJNTeHxAAKjcaLvHE3gHKardR8nTuiw=="}' 2024-09-17T15:56:24.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 01b3e74d-0aa3-49f1-8d4c-a82dbb47aec2 -i td/test-erasure-code-plugins/0/new.json 2024-09-17T15:56:24.787 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:56:24.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-17T15:56:24.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec --mkfs --key AQCopulmJNTeHxAAKjcaLvHE3gHKardR8nTuiw== --osd-uuid 01b3e74d-0aa3-49f1-8d4c-a82dbb47aec2 2024-09-17T15:56:24.829 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:56:24.827+0000 7fcef06d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:56:24.856 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:56:24.854+0000 7fcef06d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:56:24.857 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:56:24.855+0000 7fcef06d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:56:24.857 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:56:24.856+0000 7fcef06d6640 -1 bdev(0x55ebeec53c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-17T15:56:24.857 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:56:24.856+0000 7fcef06d6640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-17T15:56:30.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-17T15:56:30.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T15:56:30.508 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T15:56:30.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T15:56:30.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T15:56:31.040 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T15:56:31.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T15:56:31.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec 2024-09-17T15:56:31.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T15:56:31.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T15:56:31.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T15:56:31.067 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:56:31.065+0000 7fe5e7230640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:56:31.084 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:56:31.082+0000 7fe5e7230640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:56:31.085 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:56:31.083+0000 7fe5e7230640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:56:31.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T15:56:31.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T15:56:31.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T15:56:31.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T15:56:31.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T15:56:31.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:31.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T15:56:31.499 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:56:31.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:31.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:31.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:32.441 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:56:32.440+0000 7fe5e7230640 -1 Falling back to public interface 2024-09-17T15:56:32.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:32.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:32.926 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T15:56:32.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T15:56:32.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:32.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:33.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:34.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:34.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:34.359 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T15:56:34.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T15:56:34.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:34.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:34.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:35.553 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:56:35.551+0000 7fe5e7230640 -1 osd.0 0 log_to_monitors true 2024-09-17T15:56:35.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:35.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:35.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T15:56:35.785 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T15:56:35.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:35.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:36.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:37.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:37.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:37.297 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T15:56:37.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T15:56:37.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:37.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:37.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:38.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:38.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:38.693 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T15:56:38.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T15:56:38.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:38.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:39.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:40.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:40.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:40.111 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T15:56:40.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T15:56:40.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:40.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:40.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:41.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:41.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:41.630 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T15:56:41.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T15:56:41.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:41.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:42.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:43.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:43.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:43.028 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T15:56:43.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T15:56:43.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:43.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:43.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:44.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:44.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:44.434 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T15:56:44.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T15:56:44.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:44.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:44.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:45.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:45.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:45.847 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T15:56:45.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T15:56:45.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:45.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:46.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:47.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:47.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:47.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T15:56:47.280 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T15:56:47.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:47.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:48.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:49.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:49.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:49.106 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T15:56:49.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T15:56:49.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:49.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:49.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:50.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:50.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:50.960 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T15:56:50.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T15:56:50.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:50.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:51.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:52.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:52.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:52.694 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T15:56:52.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T15:56:52.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:52.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:53.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:54.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:54.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:54.362 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T15:56:54.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T15:56:54.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:54.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:54.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:55.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:55.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:55.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T15:56:55.898 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T15:56:55.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:55.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:56.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:56:57.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:56:57.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:56:57.607 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T15:56:57.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T15:56:57.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:56:57.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:56:58.054 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1350799801,v1:127.0.0.1:6803/1350799801] [v2:127.0.0.1:6804/1350799801,v1:127.0.0.1:6805/1350799801] exists,up 01b3e74d-0aa3-49f1-8d4c-a82dbb47aec2 2024-09-17T15:56:58.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T15:56:58.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T15:56:58.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T15:56:58.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-17T15:56:58.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T15:56:58.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T15:56:58.057 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:56:58.057 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:58.057 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:58.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-osd.0.asok 2024-09-17T15:56:58.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-17T15:56:58.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-osd.0.asok log flush 2024-09-17T15:56:58.135 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:56:58.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-17T15:56:58.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-17T15:56:58.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-17T15:56:58.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:56:58.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:56:58.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:56:58.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:56:58.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:56:58.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:56:58.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:56:58.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:56:58.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:56:58.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:56:58.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:56:58.254 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:56:58.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:56:58.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:56:58.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:56:58.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:56:58.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:56:58.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:56:58.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:56:58.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:56:58.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:56:58.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:56:58.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:56:58.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:58.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:58.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:56:58.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:56:58.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:56:58.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-17T15:56:58.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-17T15:56:58.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-17T15:56:58.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-17T15:56:58.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:56:58.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:56:58.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:56:58.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:56:58.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:56:58.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:56:58.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:56:58.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:56:58.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:56:58.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:56:58.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:56:58.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:56:58.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:56:58.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:56:58.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:56:58.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:56:58.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:56:58.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:56:58.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:56:58.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:56:58.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:56:58.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:56:58.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:56:58.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:58.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:58.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:56:58.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:56:58.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:56:58.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-17T15:56:58.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T15:56:58.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:58.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:58.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27518 2024-09-17T15:56:58.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T15:56:58.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T15:56:58.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T15:56:58.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-17T15:56:58.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=lrc 2024-09-17T15:56:58.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-17T15:56:58.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T15:56:58.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T15:56:58.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T15:56:58.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-17T15:56:58.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=lrc 2024-09-17T15:56:58.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T15:56:58.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:56:58.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:56:58.733 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:56:58.733 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:58.733 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:58.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:56:58.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=lrc 2024-09-17T15:56:58.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T15:56:58.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T15:56:58.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:56:58.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:56:58.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T15:56:58.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T15:56:58.927 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:56:58.927 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:56:58.927 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:56:58.927 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:56:58.927 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:58.928 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:58.928 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:56:58.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:56:58.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get fsid 2024-09-17T15:56:59.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T15:56:59.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:56:59.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:56:59.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T15:56:59.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T15:56:59.008 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:56:59.008 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:56:59.008 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:56:59.008 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:56:59.008 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:59.008 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:59.009 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:56:59.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:56:59.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get mon_host 2024-09-17T15:56:59.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-17T15:56:59.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-17T15:56:59.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T15:56:59.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T15:56:59.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T15:56:59.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-17T15:56:59.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T15:56:59.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T15:56:59.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:56:59.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:56:59.939 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:56:59.939 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:59.939 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:59.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:56:59.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:56:59.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:56:59.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-17T15:56:59.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:56:59.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:56:59.967 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:56:59.967 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:56:59.967 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:56:59.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:56:59.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-17T15:56:59.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T15:57:00.039 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:57:00.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=lrc 2024-09-17T15:57:00.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-17T15:57:00.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T15:57:00.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T15:57:00.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T15:57:00.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-17T15:57:00.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-17T15:57:00.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T15:57:00.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T15:57:00.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T15:57:00.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-17T15:57:00.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-17T15:57:00.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T15:57:00.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T15:57:00.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-17T15:57:00.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T15:57:00.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:57:00.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:57:00.051 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:57:00.051 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:57:00.051 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:57:00.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:57:00.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:57:00.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T15:57:00.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T15:57:00.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T15:57:00.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-17T15:57:00.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-17T15:57:00.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T15:57:00.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T15:57:00.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T15:57:00.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T15:57:00.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T15:57:00.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=lrc 2024-09-17T15:57:00.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-17T15:57:00.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T15:57:00.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f06180f1-ea60-4df4-8931-6415d82ecdd7 2024-09-17T15:57:00.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f06180f1-ea60-4df4-8931-6415d82ecdd7' 2024-09-17T15:57:00.053 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 f06180f1-ea60-4df4-8931-6415d82ecdd7 2024-09-17T15:57:00.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T15:57:00.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDMpulmtXLsAxAAhpVlhIgp+r00SoGE5hBkDA== 2024-09-17T15:57:00.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDMpulmtXLsAxAAhpVlhIgp+r00SoGE5hBkDA=="}' 2024-09-17T15:57:00.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f06180f1-ea60-4df4-8931-6415d82ecdd7 -i td/test-erasure-code-plugins/0/new.json 2024-09-17T15:57:00.325 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:57:00.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-17T15:57:00.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc --mkfs --key AQDMpulmtXLsAxAAhpVlhIgp+r00SoGE5hBkDA== --osd-uuid f06180f1-ea60-4df4-8931-6415d82ecdd7 2024-09-17T15:57:00.362 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:00.360+0000 7f5fc5e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:57:00.393 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:00.391+0000 7f5fc5e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:57:00.395 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:00.392+0000 7f5fc5e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:57:00.395 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:00.393+0000 7f5fc5e0d640 -1 bdev(0x55f67baa9c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-17T15:57:00.395 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:00.393+0000 7f5fc5e0d640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-17T15:57:05.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-17T15:57:05.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T15:57:05.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T15:57:05.855 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T15:57:05.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T15:57:06.392 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T15:57:06.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T15:57:06.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc 2024-09-17T15:57:06.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T15:57:06.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T15:57:06.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T15:57:06.418 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:06.416+0000 7f2542c20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:57:06.420 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:06.418+0000 7f2542c20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:57:06.421 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:06.419+0000 7f2542c20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:57:06.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T15:57:06.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T15:57:06.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T15:57:06.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T15:57:06.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T15:57:06.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:06.809 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:57:06.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T15:57:06.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:06.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:07.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:07.522 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:07.520+0000 7f2542c20640 -1 Falling back to public interface 2024-09-17T15:57:08.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:08.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:08.227 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T15:57:08.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T15:57:08.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:08.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:08.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:09.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:09.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:09.646 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T15:57:09.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T15:57:09.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:09.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:10.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:10.664 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:10.662+0000 7f2542c20640 -1 osd.0 0 log_to_monitors true 2024-09-17T15:57:11.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:11.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:11.088 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T15:57:11.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T15:57:11.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:11.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:11.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:12.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:12.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:12.718 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T15:57:12.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T15:57:12.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:12.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:13.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:14.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:14.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:14.124 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T15:57:14.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T15:57:14.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:14.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:14.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:15.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:15.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:15.552 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T15:57:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T15:57:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:15.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:16.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:16.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:16.949 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T15:57:16.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T15:57:16.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:16.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:17.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:18.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:18.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:18.424 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T15:57:18.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T15:57:18.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:18.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:18.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:19.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:19.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:19.852 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T15:57:19.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T15:57:19.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:19.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:20.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:21.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:21.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:21.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T15:57:21.251 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T15:57:21.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:21.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:21.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:22.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:22.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:22.963 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T15:57:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T15:57:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:23.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:24.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:24.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:24.548 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T15:57:24.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T15:57:24.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:24.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:25.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:26.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:26.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:26.267 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T15:57:26.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T15:57:26.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:26.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:27.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:28.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:28.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:28.380 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T15:57:28.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T15:57:28.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:28.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:29.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:30.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:30.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:30.115 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T15:57:30.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T15:57:30.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:30.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:30.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:31.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:31.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:31.741 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T15:57:31.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T15:57:31.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:31.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:32.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:33.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:33.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:33.397 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T15:57:33.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T15:57:33.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:33.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:33.830 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2009705346,v1:127.0.0.1:6803/2009705346] [v2:127.0.0.1:6804/2009705346,v1:127.0.0.1:6805/2009705346] exists,up f06180f1-ea60-4df4-8931-6415d82ecdd7 2024-09-17T15:57:33.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T15:57:33.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T15:57:33.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T15:57:33.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-17T15:57:33.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T15:57:33.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T15:57:33.832 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:57:33.832 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:57:33.832 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:57:33.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-osd.0.asok 2024-09-17T15:57:33.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-17T15:57:33.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-osd.0.asok log flush 2024-09-17T15:57:33.910 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:57:33.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-17T15:57:33.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-17T15:57:33.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-17T15:57:33.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:57:33.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:57:33.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:57:33.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:57:33.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:57:33.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:57:33.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:57:33.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:57:34.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:57:34.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:57:34.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:57:34.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:57:34.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:57:34.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:57:34.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:57:34.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:57:34.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:57:34.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:57:34.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:57:34.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:57:34.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:57:34.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:57:34.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:57:34.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:57:34.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:57:34.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:57:34.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:57:34.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:57:34.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-17T15:57:34.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-17T15:57:34.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-17T15:57:34.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-17T15:57:34.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:57:34.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:57:34.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:57:34.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:57:34.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:57:34.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:57:34.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:57:34.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:57:34.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:57:34.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:57:34.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:57:34.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:57:34.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:57:34.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:57:34.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:57:34.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:57:34.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:57:34.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:57:34.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:57:34.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:57:34.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:57:34.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:57:34.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:57:34.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:57:34.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:57:34.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:57:34.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:57:34.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:57:34.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-17T15:57:34.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T15:57:34.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:57:34.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:57:34.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27518 2024-09-17T15:57:34.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T15:57:34.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T15:57:34.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T15:57:34.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-17T15:57:34.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=isa 2024-09-17T15:57:34.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-17T15:57:34.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T15:57:34.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T15:57:34.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T15:57:34.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-17T15:57:34.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=isa 2024-09-17T15:57:34.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T15:57:34.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:57:34.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:57:34.559 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:57:34.559 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:57:34.559 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:57:34.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:57:34.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=isa 2024-09-17T15:57:34.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T15:57:34.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T15:57:34.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:57:34.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:57:34.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T15:57:34.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T15:57:34.761 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:57:34.761 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:57:34.761 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:57:34.761 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:57:34.762 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:57:34.762 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:57:34.762 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:57:34.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:57:34.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get fsid 2024-09-17T15:57:34.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T15:57:34.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:57:34.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:57:34.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T15:57:34.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T15:57:34.841 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:57:34.841 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:57:34.841 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:57:34.841 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:57:34.841 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:57:34.841 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:57:34.841 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:57:34.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:57:34.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get mon_host 2024-09-17T15:57:34.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-17T15:57:34.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-17T15:57:34.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T15:57:34.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T15:57:34.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T15:57:34.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-17T15:57:34.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T15:57:35.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T15:57:35.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:57:35.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:57:35.765 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:57:35.765 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:57:35.765 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:57:35.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:57:35.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:57:35.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:57:35.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-17T15:57:35.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:57:35.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:57:35.795 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:57:35.795 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:57:35.795 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:57:35.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:57:35.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-17T15:57:35.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T15:57:35.868 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:57:35.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=isa 2024-09-17T15:57:35.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-17T15:57:35.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T15:57:35.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T15:57:35.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T15:57:35.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-17T15:57:35.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-17T15:57:35.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T15:57:35.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T15:57:35.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T15:57:35.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-17T15:57:35.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-17T15:57:35.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T15:57:35.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T15:57:35.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-17T15:57:35.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T15:57:35.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:57:35.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:57:35.879 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:57:35.879 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:57:35.880 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:57:35.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:57:35.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:57:35.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T15:57:35.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T15:57:35.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T15:57:35.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-17T15:57:35.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-17T15:57:35.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T15:57:35.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T15:57:35.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T15:57:35.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T15:57:35.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T15:57:35.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=isa 2024-09-17T15:57:35.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-17T15:57:35.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T15:57:35.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=090ecead-0e14-4082-a352-93f142987c7b 2024-09-17T15:57:35.881 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 090ecead-0e14-4082-a352-93f142987c7b 2024-09-17T15:57:35.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 090ecead-0e14-4082-a352-93f142987c7b' 2024-09-17T15:57:35.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T15:57:35.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDvpulmOoVcNRAAjpGP1Wn5Jn7n9+h7TUHGbw== 2024-09-17T15:57:35.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDvpulmOoVcNRAAjpGP1Wn5Jn7n9+h7TUHGbw=="}' 2024-09-17T15:57:35.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 090ecead-0e14-4082-a352-93f142987c7b -i td/test-erasure-code-plugins/0/new.json 2024-09-17T15:57:36.175 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:57:36.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-17T15:57:36.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa --mkfs --key AQDvpulmOoVcNRAAjpGP1Wn5Jn7n9+h7TUHGbw== --osd-uuid 090ecead-0e14-4082-a352-93f142987c7b 2024-09-17T15:57:36.214 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:36.211+0000 7f293a320640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:57:36.244 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:36.241+0000 7f293a320640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:57:36.245 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:36.242+0000 7f293a320640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:57:36.245 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:36.243+0000 7f293a320640 -1 bdev(0x561a1164fc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-17T15:57:36.245 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:36.243+0000 7f293a320640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-17T15:57:41.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-17T15:57:41.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T15:57:41.488 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T15:57:41.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T15:57:41.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T15:57:42.022 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T15:57:42.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T15:57:42.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa 2024-09-17T15:57:42.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T15:57:42.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T15:57:42.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T15:57:42.047 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:42.046+0000 7f2bacc20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:57:42.070 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:42.069+0000 7f2bacc20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:57:42.071 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:42.070+0000 7f2bacc20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:57:42.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T15:57:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T15:57:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T15:57:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T15:57:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T15:57:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:42.476 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:57:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T15:57:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:42.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:43.662 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:43.660+0000 7f2bacc20640 -1 Falling back to public interface 2024-09-17T15:57:43.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:43.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:43.886 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T15:57:43.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T15:57:43.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:43.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:44.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:45.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:45.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:45.311 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T15:57:45.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T15:57:45.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:45.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:45.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:46.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:46.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:46.729 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T15:57:46.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T15:57:46.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:46.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:46.781 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:57:46.779+0000 7f2bacc20640 -1 osd.0 0 log_to_monitors true 2024-09-17T15:57:47.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:48.234 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T15:57:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T15:57:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:48.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:49.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:49.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:49.713 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T15:57:49.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T15:57:49.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:49.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:50.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:51.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:51.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:51.128 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T15:57:51.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T15:57:51.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:51.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:51.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:52.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:52.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:52.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T15:57:52.676 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T15:57:52.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:52.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:53.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:54.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:54.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:54.073 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T15:57:54.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T15:57:54.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:54.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:54.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:55.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:55.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:55.564 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T15:57:55.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T15:57:55.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:55.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:55.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:56.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:56.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:56.965 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T15:57:56.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T15:57:56.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:56.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:57.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:57:58.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:57:58.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:57:58.360 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T15:57:58.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T15:57:58.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:57:58.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:57:59.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:00.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:00.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:00.170 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T15:58:00.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T15:58:00.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:00.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:01.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:02.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:02.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:02.291 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T15:58:02.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T15:58:02.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:02.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:03.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:04.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:04.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:04.018 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T15:58:04.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T15:58:04.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:04.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:04.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:05.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:05.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:05.971 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T15:58:05.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T15:58:05.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:05.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:06.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:07.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:07.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:07.789 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T15:58:07.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T15:58:07.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:07.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:08.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:09.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:09.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:09.207 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T15:58:09.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T15:58:09.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:09.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:09.669 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1348763303,v1:127.0.0.1:6803/1348763303] [v2:127.0.0.1:6804/1348763303,v1:127.0.0.1:6805/1348763303] exists,up 090ecead-0e14-4082-a352-93f142987c7b 2024-09-17T15:58:09.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T15:58:09.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T15:58:09.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T15:58:09.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-17T15:58:09.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T15:58:09.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T15:58:09.671 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:58:09.671 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:09.671 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:09.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-osd.0.asok 2024-09-17T15:58:09.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-17T15:58:09.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-osd.0.asok log flush 2024-09-17T15:58:09.750 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:58:09.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-17T15:58:09.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-17T15:58:09.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-17T15:58:09.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:58:09.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:58:09.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:58:09.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:58:09.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:58:09.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:58:09.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:58:09.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:58:09.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:58:09.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:58:09.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:58:09.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:58:09.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:58:09.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:58:09.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:58:09.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:58:09.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:58:09.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:58:09.872 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:58:09.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:58:09.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:58:09.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:58:09.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:58:09.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:09.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:09.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:58:09.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:58:09.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:58:09.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:72: TEST_preload_no_warning: return 0 2024-09-17T15:58:09.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-17T15:58:09.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning_default td/test-erasure-code-plugins 2024-09-17T15:58:09.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:76: TEST_preload_no_warning_default: local dir=td/test-erasure-code-plugins 2024-09-17T15:58:09.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:78: TEST_preload_no_warning_default: setup td/test-erasure-code-plugins 2024-09-17T15:58:09.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-17T15:58:09.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-17T15:58:09.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:58:09.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:58:09.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:58:09.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:58:09.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:58:09.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:58:09.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:58:09.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:58:09.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:58:09.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:58:09.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:58:09.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:58:09.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:58:09.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:58:09.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:58:09.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:58:09.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:58:09.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:58:09.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:58:09.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:58:09.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:58:09.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:58:09.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:58:09.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:09.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:09.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:58:09.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:58:09.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:58:09.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-17T15:58:09.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T15:58:09.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:09.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:09.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27518 2024-09-17T15:58:09.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T15:58:09.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T15:58:09.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T15:58:09.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-17T15:58:09.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:79: TEST_preload_no_warning_default: run_mon td/test-erasure-code-plugins a 2024-09-17T15:58:09.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-17T15:58:09.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T15:58:09.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T15:58:09.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T15:58:09.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-17T15:58:09.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-09-17T15:58:10.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T15:58:10.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:58:10.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:58:10.349 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:58:10.349 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:10.349 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:10.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:58:10.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T15:58:10.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T15:58:10.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T15:58:10.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:58:10.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:58:10.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T15:58:10.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T15:58:10.534 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:58:10.534 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:58:10.534 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:58:10.534 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:58:10.534 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:10.534 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:10.534 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:58:10.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:58:10.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get fsid 2024-09-17T15:58:10.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T15:58:10.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:58:10.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:58:10.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T15:58:10.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T15:58:10.614 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:58:10.614 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:58:10.614 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:58:10.614 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:58:10.614 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:10.614 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:10.614 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:58:10.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:58:10.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get mon_host 2024-09-17T15:58:10.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: get_asok_path mon.a 2024-09-17T15:58:10.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:58:10.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:58:10.692 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:58:10.692 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:10.692 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:10.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:58:10.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: CEPH_ARGS= 2024-09-17T15:58:10.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T15:58:10.762 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:58:10.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:81: TEST_preload_no_warning_default: run_mgr td/test-erasure-code-plugins x 2024-09-17T15:58:10.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-17T15:58:10.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T15:58:10.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T15:58:10.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T15:58:10.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-17T15:58:10.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T15:58:11.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T15:58:11.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:58:11.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:58:11.605 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:58:11.606 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:11.606 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:11.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:58:11.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:58:11.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:58:11.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:82: TEST_preload_no_warning_default: run_osd td/test-erasure-code-plugins 0 2024-09-17T15:58:11.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-17T15:58:11.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T15:58:11.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T15:58:11.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T15:58:11.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-17T15:58:11.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-17T15:58:11.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T15:58:11.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T15:58:11.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T15:58:11.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-17T15:58:11.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-17T15:58:11.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T15:58:11.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T15:58:11.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-17T15:58:11.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T15:58:11.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:58:11.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:58:11.636 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:58:11.636 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:11.636 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:11.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:58:11.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:58:11.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T15:58:11.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T15:58:11.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T15:58:11.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-17T15:58:11.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-17T15:58:11.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T15:58:11.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T15:58:11.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T15:58:11.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T15:58:11.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T15:58:11.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T15:58:11.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-17T15:58:11.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T15:58:11.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6b43c3c3-9eb7-4e3a-9883-60c8c3d788aa 2024-09-17T15:58:11.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 6b43c3c3-9eb7-4e3a-9883-60c8c3d788aa' 2024-09-17T15:58:11.639 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 6b43c3c3-9eb7-4e3a-9883-60c8c3d788aa 2024-09-17T15:58:11.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T15:58:11.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQATp+lmGOfKJhAAxEBXG9zJsKP/P/AWSvXWYg== 2024-09-17T15:58:11.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQATp+lmGOfKJhAAxEBXG9zJsKP/P/AWSvXWYg=="}' 2024-09-17T15:58:11.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6b43c3c3-9eb7-4e3a-9883-60c8c3d788aa -i td/test-erasure-code-plugins/0/new.json 2024-09-17T15:58:11.915 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:58:11.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-17T15:58:11.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATp+lmGOfKJhAAxEBXG9zJsKP/P/AWSvXWYg== --osd-uuid 6b43c3c3-9eb7-4e3a-9883-60c8c3d788aa 2024-09-17T15:58:11.953 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:11.951+0000 7fee6620d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:58:11.976 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:11.974+0000 7fee6620d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:58:11.977 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:11.975+0000 7fee6620d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:58:11.977 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:11.975+0000 7fee6620d640 -1 bdev(0x564c58f79c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-17T15:58:11.977 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:11.975+0000 7fee6620d640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-17T15:58:17.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-17T15:58:17.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T15:58:17.795 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T15:58:17.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T15:58:17.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T15:58:18.321 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T15:58:18.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T15:58:18.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T15:58:18.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T15:58:18.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T15:58:18.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T15:58:18.346 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:18.343+0000 7ff68bdc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:58:18.362 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:18.359+0000 7ff68bdc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:58:18.364 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:18.361+0000 7ff68bdc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:58:18.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T15:58:18.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T15:58:18.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T15:58:18.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T15:58:18.777 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:58:18.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T15:58:18.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:18.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T15:58:18.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:18.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:19.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:19.727 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:19.724+0000 7ff68bdc3640 -1 Falling back to public interface 2024-09-17T15:58:20.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:20.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:20.203 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T15:58:20.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T15:58:20.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:20.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:20.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:21.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:21.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:21.597 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T15:58:21.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T15:58:21.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:21.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:22.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:22.856 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:22.853+0000 7ff68bdc3640 -1 osd.0 0 log_to_monitors true 2024-09-17T15:58:23.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:23.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:23.030 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T15:58:23.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T15:58:23.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:23.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:23.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:24.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:24.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:24.477 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T15:58:24.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T15:58:24.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:24.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:24.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:25.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:25.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:25.968 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T15:58:25.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T15:58:25.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:25.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:26.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:27.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:27.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:27.387 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T15:58:27.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T15:58:27.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:27.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:27.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:28.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:28.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:28.782 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T15:58:28.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T15:58:28.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:28.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:29.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:30.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:30.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:30.195 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T15:58:30.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T15:58:30.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:30.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:30.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:31.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:31.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:31.720 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T15:58:31.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T15:58:31.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:31.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:32.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:33.111 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T15:58:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T15:58:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:33.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:34.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:34.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:34.583 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T15:58:34.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T15:58:34.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:34.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:35.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:36.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:36.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:36.419 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T15:58:36.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T15:58:36.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:36.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:37.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:38.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:38.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:38.204 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T15:58:38.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T15:58:38.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:38.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:39.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:40.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:40.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:40.327 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T15:58:40.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T15:58:40.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:40.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:41.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:42.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:42.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:42.117 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T15:58:42.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T15:58:42.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:42.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:43.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:44.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:44.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:44.188 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T15:58:44.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T15:58:44.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:44.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:45.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:46.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:46.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:46.068 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T15:58:46.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T15:58:46.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:46.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:46.579 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1121477441,v1:127.0.0.1:6803/1121477441] [v2:127.0.0.1:6804/1121477441,v1:127.0.0.1:6805/1121477441] exists,up 6b43c3c3-9eb7-4e3a-9883-60c8c3d788aa 2024-09-17T15:58:46.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T15:58:46.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T15:58:46.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T15:58:46.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: get_asok_path osd.0 2024-09-17T15:58:46.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T15:58:46.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T15:58:46.582 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:58:46.582 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:46.582 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:46.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-osd.0.asok 2024-09-17T15:58:46.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: CEPH_ARGS= 2024-09-17T15:58:46.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-osd.0.asok log flush 2024-09-17T15:58:46.659 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:58:46.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:84: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/mon.a.log 2024-09-17T15:58:46.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:85: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/osd.0.log 2024-09-17T15:58:46.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:86: TEST_preload_no_warning_default: teardown td/test-erasure-code-plugins 2024-09-17T15:58:46.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:58:46.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:58:46.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:58:46.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:58:46.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:58:46.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:58:46.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:58:46.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:58:46.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:58:46.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:58:46.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:58:46.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:58:46.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:58:46.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:58:46.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:58:46.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:58:46.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:58:46.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:58:46.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:58:46.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:58:46.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:58:46.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:58:46.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:58:46.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:46.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:46.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:58:46.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:58:46.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:58:46.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:88: TEST_preload_no_warning_default: return 0 2024-09-17T15:58:46.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-17T15:58:46.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_warning td/test-erasure-code-plugins 2024-09-17T15:58:46.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:41: TEST_preload_warning: local dir=td/test-erasure-code-plugins 2024-09-17T15:58:46.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-17T15:58:46.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-17T15:58:46.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-17T15:58:46.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-17T15:58:46.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:58:46.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:58:46.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:58:46.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:58:46.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:58:46.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:58:46.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:58:46.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:58:46.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:58:46.792 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:58:46.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:58:46.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:58:46.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:58:46.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:58:46.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:58:46.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:58:46.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:58:46.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:58:46.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:58:46.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:58:46.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:58:46.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:58:46.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:58:46.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:46.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:46.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:58:46.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:58:46.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:58:46.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-17T15:58:46.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T15:58:46.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:46.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:46.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27518 2024-09-17T15:58:46.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T15:58:46.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T15:58:46.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T15:58:46.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-17T15:58:46.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_generic 2024-09-17T15:58:46.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-17T15:58:46.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T15:58:46.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T15:58:46.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T15:58:46.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-17T15:58:46.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_generic 2024-09-17T15:58:47.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T15:58:47.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:58:47.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:58:47.275 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:58:47.275 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:47.275 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:47.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:58:47.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_generic 2024-09-17T15:58:47.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T15:58:47.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T15:58:47.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:58:47.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:58:47.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T15:58:47.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T15:58:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:58:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:58:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:58:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:58:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:58:47.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:58:47.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get fsid 2024-09-17T15:58:47.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T15:58:47.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:58:47.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:58:47.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T15:58:47.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T15:58:47.563 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:58:47.563 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:58:47.563 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:58:47.563 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:58:47.563 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:47.563 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:47.563 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:58:47.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:58:47.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get mon_host 2024-09-17T15:58:47.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-17T15:58:47.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-17T15:58:47.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T15:58:47.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T15:58:47.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T15:58:47.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-17T15:58:47.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T15:58:48.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T15:58:48.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:58:48.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:58:48.490 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:58:48.490 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:48.490 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:48.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:58:48.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:58:48.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:58:48.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-17T15:58:48.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:58:48.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:58:48.517 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:58:48.517 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:48.517 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:48.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:58:48.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-17T15:58:48.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T15:58:48.589 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:58:48.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_generic 2024-09-17T15:58:48.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-17T15:58:48.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T15:58:48.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T15:58:48.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T15:58:48.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-17T15:58:48.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-17T15:58:48.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T15:58:48.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T15:58:48.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T15:58:48.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-17T15:58:48.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-17T15:58:48.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T15:58:48.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T15:58:48.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-17T15:58:48.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T15:58:48.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:58:48.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:58:48.602 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:58:48.602 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:58:48.602 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:58:48.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:58:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:58:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T15:58:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T15:58:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T15:58:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-17T15:58:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-17T15:58:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T15:58:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T15:58:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T15:58:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T15:58:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T15:58:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_generic 2024-09-17T15:58:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-17T15:58:48.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T15:58:48.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e6b3b5d3-655c-46d6-8c35-196dd80cc693 2024-09-17T15:58:48.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e6b3b5d3-655c-46d6-8c35-196dd80cc693' 2024-09-17T15:58:48.604 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 e6b3b5d3-655c-46d6-8c35-196dd80cc693 2024-09-17T15:58:48.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T15:58:48.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA4p+lm7kmsJBAAC6RYPrBnh1i02gLSwNa9yg== 2024-09-17T15:58:48.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA4p+lm7kmsJBAAC6RYPrBnh1i02gLSwNa9yg=="}' 2024-09-17T15:58:48.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e6b3b5d3-655c-46d6-8c35-196dd80cc693 -i td/test-erasure-code-plugins/0/new.json 2024-09-17T15:58:48.883 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:58:48.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-17T15:58:48.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic --mkfs --key AQA4p+lm7kmsJBAAC6RYPrBnh1i02gLSwNa9yg== --osd-uuid e6b3b5d3-655c-46d6-8c35-196dd80cc693 2024-09-17T15:58:48.921 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:48.919+0000 7f8d8fb02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:58:48.952 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:48.950+0000 7f8d8fb02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:58:48.953 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:48.951+0000 7f8d8fb02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:58:48.953 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:48.951+0000 7f8d8fb02640 -1 bdev(0x558279fb9c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-17T15:58:48.953 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:48.951+0000 7f8d8fb02640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-17T15:58:54.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-17T15:58:54.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T15:58:54.460 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T15:58:54.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T15:58:54.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T15:58:54.993 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T15:58:54.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T15:58:54.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic 2024-09-17T15:58:54.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T15:58:54.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T15:58:54.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T15:58:55.018 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:55.017+0000 7f55896e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:58:55.030 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:55.028+0000 7f55896e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:58:55.031 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:55.029+0000 7f55896e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:58:55.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T15:58:55.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T15:58:55.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T15:58:55.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T15:58:55.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T15:58:55.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:55.445 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:58:55.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T15:58:55.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:55.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:55.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:56.129 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:56.127+0000 7f55896e6640 -1 Falling back to public interface 2024-09-17T15:58:56.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:56.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:56.855 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T15:58:56.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T15:58:56.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:56.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:57.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:58.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:58.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:58.280 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T15:58:58.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T15:58:58.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:58.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:58:58.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:58:59.199 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:58:59.198+0000 7f55896e6640 -1 osd.0 0 log_to_monitors true 2024-09-17T15:58:59.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:58:59.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:58:59.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T15:58:59.681 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T15:58:59.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:58:59.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:00.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:01.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:01.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:01.169 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T15:59:01.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T15:59:01.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:01.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:01.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:02.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:02.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:02.561 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T15:59:02.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T15:59:02.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:02.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:03.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:04.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:04.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:04.068 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T15:59:04.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T15:59:04.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:04.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:04.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:05.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:05.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:05.462 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T15:59:05.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T15:59:05.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:05.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:05.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:06.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:06.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:06.943 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T15:59:06.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T15:59:06.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:06.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:07.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:08.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:08.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:08.343 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T15:59:08.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T15:59:08.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:08.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:08.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:09.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:09.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:09.739 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T15:59:09.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T15:59:09.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:09.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:10.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:11.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:11.228 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T15:59:11.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:11.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T15:59:11.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:11.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:11.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:12.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:12.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:12.799 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T15:59:12.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T15:59:12.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:12.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:13.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:14.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:14.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:14.427 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T15:59:14.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T15:59:14.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:14.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:14.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:16.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:16.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:16.001 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T15:59:16.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T15:59:16.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:16.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:16.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:17.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:17.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:17.662 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T15:59:17.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T15:59:17.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:17.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:18.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:19.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:19.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:19.457 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T15:59:19.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T15:59:19.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:19.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:20.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:21.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:21.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:21.267 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T15:59:21.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T15:59:21.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:21.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:21.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:22.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:22.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:22.707 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T15:59:22.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T15:59:22.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:22.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:23.127 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1769682503,v1:127.0.0.1:6803/1769682503] [v2:127.0.0.1:6804/1769682503,v1:127.0.0.1:6805/1769682503] exists,up e6b3b5d3-655c-46d6-8c35-196dd80cc693 2024-09-17T15:59:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T15:59:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T15:59:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T15:59:23.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-17T15:59:23.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T15:59:23.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T15:59:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:59:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:59:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:59:23.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-osd.0.asok 2024-09-17T15:59:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-17T15:59:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-osd.0.asok log flush 2024-09-17T15:59:23.208 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:59:23.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-17T15:59:23.218 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:58:47.305+0000 7fb314327c80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-17T15:59:23.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/osd.0.log 2024-09-17T15:59:23.219 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:58:56.128+0000 7f55896e6640 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-17T15:59:23.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-17T15:59:23.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:59:23.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:59:23.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:59:23.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:59:23.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:59:23.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:59:23.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:59:23.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:59:23.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:59:23.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:59:23.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:59:23.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:59:23.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:59:23.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:59:23.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:59:23.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:59:23.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:59:23.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:59:23.332 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:59:23.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:59:23.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:59:23.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:59:23.337 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:59:23.337 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:59:23.337 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:59:23.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:59:23.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:59:23.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:59:23.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-17T15:59:23.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-17T15:59:23.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-17T15:59:23.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-17T15:59:23.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:59:23.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:59:23.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:59:23.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:59:23.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:59:23.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:59:23.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:59:23.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:59:23.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:59:23.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:59:23.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:59:23.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:59:23.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:59:23.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:59:23.342 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:59:23.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:59:23.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:59:23.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:59:23.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:59:23.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:59:23.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:59:23.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:59:23.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:59:23.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:59:23.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:59:23.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:59:23.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:59:23.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:59:23.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-17T15:59:23.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T15:59:23.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:59:23.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:59:23.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27518 2024-09-17T15:59:23.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T15:59:23.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T15:59:23.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T15:59:23.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-17T15:59:23.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse3 2024-09-17T15:59:23.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-17T15:59:23.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T15:59:23.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T15:59:23.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T15:59:23.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-17T15:59:23.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse3 2024-09-17T15:59:23.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T15:59:23.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:59:23.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:59:23.807 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:59:23.807 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:59:23.807 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:59:23.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:59:23.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse3 2024-09-17T15:59:23.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T15:59:23.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T15:59:23.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:59:23.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:59:23.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T15:59:23.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T15:59:23.993 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:59:23.993 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:59:23.993 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:59:23.993 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:59:23.993 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:59:23.993 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:59:23.993 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:59:23.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:59:23.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get fsid 2024-09-17T15:59:24.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T15:59:24.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T15:59:24.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T15:59:24.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T15:59:24.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T15:59:24.077 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T15:59:24.077 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:59:24.077 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:59:24.077 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:59:24.077 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:59:24.078 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:59:24.078 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:59:24.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T15:59:24.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get mon_host 2024-09-17T15:59:24.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-17T15:59:24.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-17T15:59:24.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T15:59:24.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T15:59:24.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T15:59:24.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-17T15:59:24.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T15:59:25.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T15:59:25.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:59:25.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:59:25.007 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:59:25.007 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:59:25.007 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:59:25.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:59:25.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:59:25.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T15:59:25.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-17T15:59:25.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T15:59:25.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T15:59:25.035 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:59:25.035 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:59:25.035 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:59:25.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T15:59:25.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-17T15:59:25.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T15:59:25.121 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:59:25.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse3 2024-09-17T15:59:25.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-17T15:59:25.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T15:59:25.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T15:59:25.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T15:59:25.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-17T15:59:25.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-17T15:59:25.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T15:59:25.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T15:59:25.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T15:59:25.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-17T15:59:25.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-17T15:59:25.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T15:59:25.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T15:59:25.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-17T15:59:25.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T15:59:25.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:59:25.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:59:25.132 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:59:25.132 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:59:25.132 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:59:25.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:59:25.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:59:25.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T15:59:25.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T15:59:25.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T15:59:25.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-17T15:59:25.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-17T15:59:25.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T15:59:25.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T15:59:25.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T15:59:25.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T15:59:25.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T15:59:25.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse3 2024-09-17T15:59:25.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-17T15:59:25.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T15:59:25.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=29515e8b-5daa-4de6-b03e-34319a7fea0b 2024-09-17T15:59:25.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 29515e8b-5daa-4de6-b03e-34319a7fea0b' 2024-09-17T15:59:25.134 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 29515e8b-5daa-4de6-b03e-34319a7fea0b 2024-09-17T15:59:25.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T15:59:25.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBdp+lmOA+1CBAAGyDyxvpOGJf/0eg0vu0dzw== 2024-09-17T15:59:25.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBdp+lmOA+1CBAAGyDyxvpOGJf/0eg0vu0dzw=="}' 2024-09-17T15:59:25.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 29515e8b-5daa-4de6-b03e-34319a7fea0b -i td/test-erasure-code-plugins/0/new.json 2024-09-17T15:59:25.399 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:59:25.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-17T15:59:25.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 --mkfs --key AQBdp+lmOA+1CBAAGyDyxvpOGJf/0eg0vu0dzw== --osd-uuid 29515e8b-5daa-4de6-b03e-34319a7fea0b 2024-09-17T15:59:25.437 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:59:25.435+0000 7f5a3ec9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:59:25.468 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:59:25.466+0000 7f5a3ec9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:59:25.469 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:59:25.467+0000 7f5a3ec9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:59:25.469 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:59:25.468+0000 7f5a3ec9c640 -1 bdev(0x5607a5fe7c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-17T15:59:25.469 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:59:25.468+0000 7f5a3ec9c640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-17T15:59:31.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-17T15:59:31.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T15:59:31.204 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T15:59:31.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T15:59:31.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T15:59:31.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T15:59:31.742 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T15:59:31.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 2024-09-17T15:59:31.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T15:59:31.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T15:59:31.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T15:59:31.768 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:59:31.766+0000 7f0645418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:59:31.782 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:59:31.780+0000 7f0645418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:59:31.783 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:59:31.781+0000 7f0645418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T15:59:32.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T15:59:32.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T15:59:32.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T15:59:32.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T15:59:32.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T15:59:32.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:32.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T15:59:32.161 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T15:59:32.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:32.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:32.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:32.901 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:59:32.899+0000 7f0645418640 -1 Falling back to public interface 2024-09-17T15:59:33.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:33.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:33.548 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T15:59:33.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T15:59:33.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:33.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:33.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:34.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:34.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:34.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T15:59:34.941 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T15:59:34.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:34.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:35.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:35.769 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T15:59:35.767+0000 7f0645418640 -1 osd.0 0 log_to_monitors true 2024-09-17T15:59:36.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:36.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:36.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T15:59:36.340 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T15:59:36.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:36.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:36.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:37.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:37.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:37.844 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T15:59:37.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T15:59:37.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:37.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:38.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:39.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:39.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:39.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T15:59:39.246 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T15:59:39.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:39.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:39.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:40.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:40.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:40.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T15:59:40.661 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T15:59:40.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:40.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:41.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:42.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:42.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:42.059 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T15:59:42.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T15:59:42.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:42.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:42.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:43.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:43.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:43.533 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T15:59:43.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T15:59:43.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:43.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:43.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:44.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:44.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:44.927 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T15:59:44.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T15:59:44.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:44.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:45.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:46.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:46.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:46.343 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T15:59:46.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T15:59:46.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:46.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:46.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:47.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:47.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:47.885 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T15:59:47.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T15:59:47.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:47.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:48.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:49.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:49.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:49.429 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T15:59:49.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T15:59:49.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:49.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:50.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:51.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:51.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:51.356 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T15:59:51.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T15:59:51.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:51.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:52.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:53.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:53.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:53.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T15:59:53.303 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T15:59:53.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:53.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:53.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:54.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:54.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:54.980 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T15:59:54.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T15:59:54.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:54.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:55.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:56.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:56.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:56.841 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T15:59:56.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T15:59:56.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:56.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:57.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T15:59:58.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T15:59:58.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T15:59:58.752 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T15:59:58.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T15:59:58.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T15:59:58.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T15:59:59.180 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3425817933,v1:127.0.0.1:6803/3425817933] [v2:127.0.0.1:6804/3425817933,v1:127.0.0.1:6805/3425817933] exists,up 29515e8b-5daa-4de6-b03e-34319a7fea0b 2024-09-17T15:59:59.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T15:59:59.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T15:59:59.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T15:59:59.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-17T15:59:59.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T15:59:59.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T15:59:59.182 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T15:59:59.182 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:59:59.182 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:59:59.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-osd.0.asok 2024-09-17T15:59:59.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-17T15:59:59.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-osd.0.asok log flush 2024-09-17T15:59:59.261 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T15:59:59.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-17T15:59:59.271 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:59:23.835+0000 7fd02c019c80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-17T15:59:59.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/osd.0.log 2024-09-17T15:59:59.272 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:59:32.900+0000 7f0645418640 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-17T15:59:59.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-17T15:59:59.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:59:59.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:59:59.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:59:59.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:59:59.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:59:59.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:59:59.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:59:59.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:59:59.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:59:59.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:59:59.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:59:59.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:59:59.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:59:59.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:59:59.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:59:59.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:59:59.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:59:59.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:59:59.383 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:59:59.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:59:59.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:59:59.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:59:59.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:59:59.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:59:59.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:59:59.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:59:59.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:59:59.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:59:59.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-17T15:59:59.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-17T15:59:59.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-17T15:59:59.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-17T15:59:59.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T15:59:59.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T15:59:59.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T15:59:59.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T15:59:59.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T15:59:59.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T15:59:59.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T15:59:59.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T15:59:59.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T15:59:59.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T15:59:59.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T15:59:59.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T15:59:59.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T15:59:59.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T15:59:59.397 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T15:59:59.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:59:59.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T15:59:59.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T15:59:59.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:59:59.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:59:59.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T15:59:59.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T15:59:59.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T15:59:59.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:59:59.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:59:59.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T15:59:59.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T15:59:59.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T15:59:59.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-17T15:59:59.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T15:59:59.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:59:59.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:59:59.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27518 2024-09-17T15:59:59.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T15:59:59.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T15:59:59.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T15:59:59.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-17T15:59:59.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse4 2024-09-17T15:59:59.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-17T15:59:59.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T15:59:59.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T15:59:59.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T15:59:59.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-17T15:59:59.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse4 2024-09-17T15:59:59.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T15:59:59.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T15:59:59.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T15:59:59.828 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T15:59:59.828 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T15:59:59.828 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T15:59:59.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T15:59:59.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse4 2024-09-17T16:00:00.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:00:00.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:00:00.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:00:00.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:00:00.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:00:00.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:00:00.030 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:00:00.030 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:00:00.031 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:00:00.031 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:00:00.031 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:00:00.031 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:00:00.031 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T16:00:00.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:00:00.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get fsid 2024-09-17T16:00:00.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:00:00.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:00:00.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:00:00.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:00:00.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:00:00.111 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:00:00.111 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:00:00.111 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:00:00.112 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:00:00.112 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:00:00.112 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:00:00.112 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T16:00:00.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:00:00.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get mon_host 2024-09-17T16:00:00.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-17T16:00:00.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-17T16:00:00.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:00:00.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:00:00.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:00:00.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-17T16:00:00.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:00:01.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:00:01.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:00:01.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:00:01.034 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:00:01.034 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:00:01.034 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:00:01.034 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T16:00:01.034 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:00:01.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:00:01.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-17T16:00:01.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:00:01.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:00:01.062 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:00:01.062 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:00:01.062 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:00:01.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T16:00:01.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-17T16:00:01.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T16:00:01.134 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:00:01.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse4 2024-09-17T16:00:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-17T16:00:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:00:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:00:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:00:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-17T16:00:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-17T16:00:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:00:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:00:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:00:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-17T16:00:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-17T16:00:01.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:00:01.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:00:01.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-17T16:00:01.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:00:01.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:00:01.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:00:01.147 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:00:01.147 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:00:01.147 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:00:01.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T16:00:01.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T16:00:01.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:00:01.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:00:01.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:00:01.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-17T16:00:01.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-17T16:00:01.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:00:01.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:00:01.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:00:01.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:00:01.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:00:01.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse4 2024-09-17T16:00:01.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-17T16:00:01.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:00:01.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4d094863-cc64-42de-88c8-917a086423b4 2024-09-17T16:00:01.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 4d094863-cc64-42de-88c8-917a086423b4' 2024-09-17T16:00:01.149 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 4d094863-cc64-42de-88c8-917a086423b4 2024-09-17T16:00:01.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:00:01.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCBp+lm69OnCRAAJAraLPVC/AZCnFicYMfr4Q== 2024-09-17T16:00:01.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCBp+lm69OnCRAAJAraLPVC/AZCnFicYMfr4Q=="}' 2024-09-17T16:00:01.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4d094863-cc64-42de-88c8-917a086423b4 -i td/test-erasure-code-plugins/0/new.json 2024-09-17T16:00:01.420 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:00:01.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-17T16:00:01.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 --mkfs --key AQCBp+lm69OnCRAAJAraLPVC/AZCnFicYMfr4Q== --osd-uuid 4d094863-cc64-42de-88c8-917a086423b4 2024-09-17T16:00:01.459 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:01.456+0000 7f6b69e44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:00:01.489 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:01.486+0000 7f6b69e44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:00:01.490 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:01.487+0000 7f6b69e44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:00:01.490 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:01.487+0000 7f6b69e44640 -1 bdev(0x555a7dc43c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:00:01.490 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:01.487+0000 7f6b69e44640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-17T16:00:07.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-17T16:00:07.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:00:07.190 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:00:07.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:00:07.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:00:07.732 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:00:07.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:00:07.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 2024-09-17T16:00:07.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:00:07.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:00:07.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:00:07.758 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:07.756+0000 7f5678934640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:00:07.769 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:07.767+0000 7f5678934640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:00:07.770 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:07.768+0000 7f5678934640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:00:08.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:00:08.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:00:08.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:00:08.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:00:08.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:00:08.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:08.184 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:00:08.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:00:08.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:08.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:08.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:09.357 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:09.355+0000 7f5678934640 -1 Falling back to public interface 2024-09-17T16:00:09.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:09.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:09.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:00:09.602 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:00:09.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:09.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:10.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:11.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:11.028 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:00:11.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:11.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:00:11.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:11.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:11.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:12.089 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:12.087+0000 7f5678934640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:00:12.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:12.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:12.454 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:00:12.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:00:12.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:12.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:12.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:13.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:13.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:13.845 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:00:13.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:00:13.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:13.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:14.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:15.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:15.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:15.244 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:00:15.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:00:15.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:15.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:15.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:16.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:16.659 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:00:16.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:16.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:00:16.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:16.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:17.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:18.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:18.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:18.177 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:00:18.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:00:18.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:18.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:18.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:19.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:19.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:19.630 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:00:19.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:00:19.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:19.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:20.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:21.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:21.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:21.045 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:00:21.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:00:21.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:21.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:21.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:22.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:22.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:22.475 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:00:22.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:00:22.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:22.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:23.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:24.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:24.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:24.016 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:00:24.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:00:24.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:24.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:24.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:25.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:25.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:25.408 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:00:25.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:00:25.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:25.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:26.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:27.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:27.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:27.352 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:00:27.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:00:27.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:27.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:28.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:29.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:29.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:29.100 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:00:29.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:00:29.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:29.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:29.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:30.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:30.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:30.535 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:00:30.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:00:30.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:30.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:31.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:32.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:32.172 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:00:32.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:32.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:00:32.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:32.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:33.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:34.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:34.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:34.362 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:00:34.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:00:34.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:34.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:34.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:35.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:35.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:35.847 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:00:35.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:00:35.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:35.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:36.276 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1610576711,v1:127.0.0.1:6803/1610576711] [v2:127.0.0.1:6804/1610576711,v1:127.0.0.1:6805/1610576711] exists,up 4d094863-cc64-42de-88c8-917a086423b4 2024-09-17T16:00:36.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:00:36.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:00:36.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:00:36.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-17T16:00:36.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:00:36.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:00:36.279 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:00:36.279 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:00:36.279 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:00:36.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-osd.0.asok 2024-09-17T16:00:36.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-17T16:00:36.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-osd.0.asok log flush 2024-09-17T16:00:36.358 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:00:36.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-17T16:00:36.368 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T15:59:59.853+0000 7fbeba31cc80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-17T16:00:36.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/osd.0.log 2024-09-17T16:00:36.369 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:00:09.356+0000 7f5678934640 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-17T16:00:36.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-17T16:00:36.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T16:00:36.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:00:36.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T16:00:36.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:00:36.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:00:36.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:00:36.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:00:36.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:00:36.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:00:36.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:00:36.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:00:36.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:00:36.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:00:36.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:00:36.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:00:36.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:00:36.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:00:36.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:00:36.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:00:36.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:00:36.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:00:36.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T16:00:36.486 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:00:36.486 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:00:36.487 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:00:36.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T16:00:36.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:00:36.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:00:36.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-17T16:00:36.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-17T16:00:36.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-17T16:00:36.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-17T16:00:36.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T16:00:36.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:00:36.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T16:00:36.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:00:36.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:00:36.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:00:36.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:00:36.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:00:36.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:00:36.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:00:36.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:00:36.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:00:36.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:00:36.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:00:36.492 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:00:36.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:00:36.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:00:36.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:00:36.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:00:36.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:00:36.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:00:36.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T16:00:36.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:00:36.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:00:36.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:00:36.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T16:00:36.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:00:36.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:00:36.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-17T16:00:36.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:00:36.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:00:36.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:00:36.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27518 2024-09-17T16:00:36.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:00:36.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T16:00:36.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:00:36.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-17T16:00:36.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_generic 2024-09-17T16:00:36.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-17T16:00:36.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:00:36.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:00:36.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:00:36.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-17T16:00:36.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_generic 2024-09-17T16:00:36.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:00:36.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:00:36.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:00:36.913 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:00:36.914 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:00:36.914 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:00:36.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T16:00:36.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_generic 2024-09-17T16:00:37.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:00:37.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:00:37.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:00:37.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:00:37.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:00:37.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:00:37.108 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:00:37.108 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:00:37.108 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:00:37.108 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:00:37.108 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:00:37.108 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:00:37.108 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T16:00:37.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:00:37.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get fsid 2024-09-17T16:00:37.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:00:37.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:00:37.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:00:37.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:00:37.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:00:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:00:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:00:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:00:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:00:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:00:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:00:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T16:00:37.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:00:37.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get mon_host 2024-09-17T16:00:37.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-17T16:00:37.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-17T16:00:37.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:00:37.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:00:37.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:00:37.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-17T16:00:37.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:00:38.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:00:38.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:00:38.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:00:38.112 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:00:38.112 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:00:38.112 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:00:38.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T16:00:38.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:00:38.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:00:38.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-17T16:00:38.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:00:38.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:00:38.140 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:00:38.140 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:00:38.140 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:00:38.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T16:00:38.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-17T16:00:38.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T16:00:38.212 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:00:38.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_generic 2024-09-17T16:00:38.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-17T16:00:38.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:00:38.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:00:38.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:00:38.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-17T16:00:38.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-17T16:00:38.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:00:38.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:00:38.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:00:38.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-17T16:00:38.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-17T16:00:38.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:00:38.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:00:38.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-17T16:00:38.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:00:38.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:00:38.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:00:38.225 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:00:38.225 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:00:38.225 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:00:38.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T16:00:38.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T16:00:38.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:00:38.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:00:38.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:00:38.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-17T16:00:38.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-17T16:00:38.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:00:38.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:00:38.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:00:38.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:00:38.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:00:38.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_generic 2024-09-17T16:00:38.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-17T16:00:38.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:00:38.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c916963d-12d5-4e62-aaba-3fc6d51aed93 2024-09-17T16:00:38.226 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 c916963d-12d5-4e62-aaba-3fc6d51aed93 2024-09-17T16:00:38.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c916963d-12d5-4e62-aaba-3fc6d51aed93' 2024-09-17T16:00:38.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:00:38.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCmp+lml9U2DhAASRMnHOJuwX8ICHufeotxFA== 2024-09-17T16:00:38.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCmp+lml9U2DhAASRMnHOJuwX8ICHufeotxFA=="}' 2024-09-17T16:00:38.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c916963d-12d5-4e62-aaba-3fc6d51aed93 -i td/test-erasure-code-plugins/0/new.json 2024-09-17T16:00:38.489 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:00:38.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-17T16:00:38.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic --mkfs --key AQCmp+lml9U2DhAASRMnHOJuwX8ICHufeotxFA== --osd-uuid c916963d-12d5-4e62-aaba-3fc6d51aed93 2024-09-17T16:00:38.527 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:38.525+0000 7f67e22f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:00:38.549 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:38.547+0000 7f67e22f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:00:38.550 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:38.548+0000 7f67e22f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:00:38.551 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:38.549+0000 7f67e22f1640 -1 bdev(0x5573d497dc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:00:38.551 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:38.549+0000 7f67e22f1640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-17T16:00:44.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-17T16:00:44.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:00:44.306 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:00:44.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:00:44.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:00:44.840 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:00:44.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:00:44.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic 2024-09-17T16:00:44.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:00:44.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:00:44.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:00:44.866 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:44.863+0000 7f2c63012640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:00:44.877 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:44.874+0000 7f2c63012640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:00:44.878 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:44.875+0000 7f2c63012640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:00:45.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:00:45.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:00:45.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:00:45.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:00:45.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:00:45.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:45.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:00:45.288 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:00:45.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:45.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:45.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:45.994 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:45.992+0000 7f2c63012640 -1 Falling back to public interface 2024-09-17T16:00:46.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:46.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:46.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:00:46.715 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:00:46.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:46.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:47.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:48.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:48.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:48.134 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:00:48.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:00:48.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:48.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:48.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:49.079 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:00:49.076+0000 7f2c63012640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:00:49.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:49.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:49.529 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:00:49.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:00:49.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:49.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:50.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:51.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:51.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:51.112 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:00:51.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:00:51.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:51.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:51.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:52.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:52.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:52.522 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:00:52.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:00:52.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:52.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:52.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:53.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:53.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:53.985 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:00:53.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:00:53.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:53.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:54.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:55.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:55.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:55.412 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:00:55.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:00:55.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:55.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:55.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:56.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:56.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:56.837 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:00:56.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:00:56.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:56.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:57.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:58.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:58.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:58.234 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:00:58.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:00:58.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:58.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:00:58.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:00:59.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:00:59.649 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:00:59.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:00:59.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:00:59.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:00:59.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:00.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:01.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:01.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:01.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:01:01.166 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:01:01.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:01.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:01.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:02.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:02.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:02.818 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:01:02.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:01:02.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:02.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:03.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:04.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:04.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:04.562 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:01:04.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:01:04.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:04.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:05.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:06.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:06.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:06.583 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:01:06.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:01:06.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:06.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:07.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:08.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:08.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:08.025 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:01:08.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:01:08.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:08.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:08.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:09.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:09.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:09.480 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:01:09.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:01:09.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:09.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:10.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:11.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:11.433 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:01:11.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:11.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:01:11.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:11.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:11.852 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2249557948,v1:127.0.0.1:6803/2249557948] [v2:127.0.0.1:6804/2249557948,v1:127.0.0.1:6805/2249557948] exists,up c916963d-12d5-4e62-aaba-3fc6d51aed93 2024-09-17T16:01:11.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:01:11.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:01:11.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:01:11.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-17T16:01:11.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:01:11.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:01:11.854 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:01:11.854 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:11.854 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:11.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-osd.0.asok 2024-09-17T16:01:11.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-17T16:01:11.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-osd.0.asok log flush 2024-09-17T16:01:11.932 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:01:11.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-17T16:01:11.942 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:00:36.941+0000 7f8a66c0cc80 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-17T16:01:11.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/osd.0.log 2024-09-17T16:01:11.943 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:00:45.993+0000 7f2c63012640 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-17T16:01:11.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-17T16:01:11.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T16:01:11.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:01:11.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T16:01:11.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:01:11.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:01:11.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:01:11.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:01:11.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:01:12.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:01:12.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:01:12.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:01:12.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:01:12.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:01:12.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:01:12.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:01:12.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:01:12.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:01:12.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:01:12.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:01:12.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:01:12.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:01:12.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T16:01:12.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:01:12.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:12.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:12.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T16:01:12.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:01:12.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:01:12.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-17T16:01:12.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-17T16:01:12.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-17T16:01:12.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-17T16:01:12.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T16:01:12.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:01:12.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T16:01:12.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:01:12.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:01:12.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:01:12.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:01:12.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:01:12.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:01:12.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:01:12.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:01:12.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:01:12.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:01:12.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:01:12.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:01:12.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:01:12.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:01:12.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:01:12.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:01:12.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:01:12.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:01:12.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T16:01:12.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:01:12.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:12.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:12.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T16:01:12.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:01:12.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:01:12.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-17T16:01:12.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:01:12.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:12.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:12.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27518 2024-09-17T16:01:12.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:01:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T16:01:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:01:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-17T16:01:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse3 2024-09-17T16:01:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-17T16:01:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:01:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:01:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:01:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-17T16:01:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse3 2024-09-17T16:01:12.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:01:12.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:01:12.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:01:12.532 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:01:12.532 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:12.532 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:12.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T16:01:12.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse3 2024-09-17T16:01:12.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:01:12.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:01:12.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:01:12.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:01:12.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:01:12.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:01:12.719 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:01:12.719 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:01:12.720 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:01:12.720 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:01:12.720 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:12.720 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:12.720 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T16:01:12.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:01:12.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get fsid 2024-09-17T16:01:12.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:01:12.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:01:12.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:01:12.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:01:12.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:01:12.799 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:01:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:01:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:01:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:01:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T16:01:12.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:01:12.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get mon_host 2024-09-17T16:01:12.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-17T16:01:12.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-17T16:01:12.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:01:12.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:01:12.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:01:12.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-17T16:01:12.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:01:13.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:01:13.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:01:13.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:01:13.722 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:01:13.722 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:13.722 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:13.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T16:01:13.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:01:13.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:01:13.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-17T16:01:13.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:01:13.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:01:13.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:01:13.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:13.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:13.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T16:01:13.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-17T16:01:13.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T16:01:13.838 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:01:13.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse3 2024-09-17T16:01:13.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-17T16:01:13.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:01:13.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:01:13.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:01:13.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-17T16:01:13.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-17T16:01:13.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:01:13.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:01:13.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:01:13.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-17T16:01:13.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-17T16:01:13.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:01:13.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:01:13.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-17T16:01:13.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:01:13.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:01:13.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:01:13.850 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:01:13.850 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:13.850 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:13.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T16:01:13.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T16:01:13.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:01:13.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:01:13.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:01:13.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-17T16:01:13.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-17T16:01:13.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:01:13.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:01:13.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:01:13.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:01:13.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:01:13.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse3 2024-09-17T16:01:13.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-17T16:01:13.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:01:13.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3f195639-e7e6-45ed-92a6-d532d71b42cf 2024-09-17T16:01:13.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3f195639-e7e6-45ed-92a6-d532d71b42cf' 2024-09-17T16:01:13.852 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 3f195639-e7e6-45ed-92a6-d532d71b42cf 2024-09-17T16:01:13.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:01:13.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDJp+lmtr2MMxAAjhBYTnky4jktLKt8RHcp9w== 2024-09-17T16:01:13.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDJp+lmtr2MMxAAjhBYTnky4jktLKt8RHcp9w=="}' 2024-09-17T16:01:13.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3f195639-e7e6-45ed-92a6-d532d71b42cf -i td/test-erasure-code-plugins/0/new.json 2024-09-17T16:01:14.151 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:01:14.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-17T16:01:14.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 --mkfs --key AQDJp+lmtr2MMxAAjhBYTnky4jktLKt8RHcp9w== --osd-uuid 3f195639-e7e6-45ed-92a6-d532d71b42cf 2024-09-17T16:01:14.189 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:14.187+0000 7feb27a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:01:14.192 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:14.191+0000 7feb27a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:01:14.193 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:14.192+0000 7feb27a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:01:14.194 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:14.192+0000 7feb27a11640 -1 bdev(0x557168599c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:01:14.194 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:14.192+0000 7feb27a11640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-17T16:01:19.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-17T16:01:19.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:01:19.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:01:19.644 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:01:19.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:01:20.187 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:01:20.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:01:20.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 2024-09-17T16:01:20.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:01:20.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:01:20.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:01:20.213 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:20.211+0000 7ffa841dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:01:20.214 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:20.213+0000 7ffa841dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:01:20.215 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:20.214+0000 7ffa841dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:01:20.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:01:20.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:01:20.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:01:20.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:01:20.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:01:20.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:20.609 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:01:20.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:01:20.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:20.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:21.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:21.072 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:21.070+0000 7ffa841dd640 -1 Falling back to public interface 2024-09-17T16:01:22.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:22.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:22.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:01:22.035 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:01:22.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:22.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:22.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:23.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:23.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:23.470 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:01:23.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:01:23.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:23.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:23.704 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:23.702+0000 7ffa841dd640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:01:23.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:24.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:24.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:24.881 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:01:24.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:01:24.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:24.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:25.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:26.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:26.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:26.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:01:26.295 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:01:26.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:26.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:26.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:27.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:27.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:27.777 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:01:27.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:01:27.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:27.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:28.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:29.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:29.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:29.169 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:01:29.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:01:29.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:29.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:29.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:30.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:30.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:30.668 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:01:30.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:01:30.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:30.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:31.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:32.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:32.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:32.070 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:01:32.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:01:32.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:32.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:32.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:33.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:33.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:33.568 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:01:33.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:01:33.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:33.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:33.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:34.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:34.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:34.983 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:01:34.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:01:34.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:34.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:35.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:36.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:36.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:36.512 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:01:36.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:01:36.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:36.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:36.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:37.970 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:01:37.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:37.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:01:37.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:37.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:39.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:40.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:40.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:40.096 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:01:40.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:01:40.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:40.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:40.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:41.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:41.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:41.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:01:41.889 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:01:41.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:41.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:43.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:43.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:43.999 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:01:43.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:01:43.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:43.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:44.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:45.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:45.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:45.829 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:01:45.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:01:45.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:45.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:46.270 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2875366201,v1:127.0.0.1:6803/2875366201] [v2:127.0.0.1:6804/2875366201,v1:127.0.0.1:6805/2875366201] exists,up 3f195639-e7e6-45ed-92a6-d532d71b42cf 2024-09-17T16:01:46.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:01:46.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:01:46.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:01:46.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-17T16:01:46.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:01:46.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:01:46.272 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:01:46.273 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:46.273 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:46.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-osd.0.asok 2024-09-17T16:01:46.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-17T16:01:46.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-osd.0.asok log flush 2024-09-17T16:01:46.352 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:01:46.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-17T16:01:46.362 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:01:12.559+0000 7fadbb60ec80 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-17T16:01:46.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/osd.0.log 2024-09-17T16:01:46.363 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:01:21.071+0000 7ffa841dd640 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-17T16:01:46.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-17T16:01:46.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T16:01:46.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:01:46.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T16:01:46.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:01:46.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:01:46.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:01:46.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:01:46.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:01:46.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:01:46.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:01:46.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:01:46.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:01:46.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:01:46.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:01:46.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:01:46.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:01:46.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:01:46.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:01:46.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:01:46.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:01:46.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:01:46.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T16:01:46.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:01:46.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:46.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:46.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T16:01:46.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:01:46.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:01:46.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-17T16:01:46.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-17T16:01:46.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-17T16:01:46.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-17T16:01:46.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T16:01:46.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:01:46.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T16:01:46.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:01:46.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:01:46.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:01:46.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:01:46.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:01:46.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:01:46.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:01:46.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:01:46.485 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:01:46.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:01:46.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:01:46.486 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:01:46.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:01:46.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:01:46.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:01:46.488 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:01:46.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:01:46.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:01:46.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T16:01:46.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:01:46.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:46.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:46.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T16:01:46.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:01:46.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:01:46.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-17T16:01:46.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:01:46.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:46.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:46.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27518 2024-09-17T16:01:46.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:01:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T16:01:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:01:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-17T16:01:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse4 2024-09-17T16:01:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-17T16:01:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:01:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:01:46.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:01:46.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-17T16:01:46.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse4 2024-09-17T16:01:46.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:01:46.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:01:46.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:01:46.951 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:01:46.951 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:46.951 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:46.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T16:01:46.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse4 2024-09-17T16:01:47.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:01:47.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:01:47.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:01:47.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:01:47.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:01:47.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:01:47.145 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:01:47.145 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:01:47.145 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:01:47.146 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:01:47.146 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:47.146 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:47.146 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T16:01:47.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:01:47.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get fsid 2024-09-17T16:01:47.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:01:47.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:01:47.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:01:47.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:01:47.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:01:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:01:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:01:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:01:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:01:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T16:01:47.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:01:47.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27518/ceph-mon.a.asok config get mon_host 2024-09-17T16:01:47.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-17T16:01:47.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-17T16:01:47.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:01:47.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:01:47.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:01:47.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-17T16:01:47.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:01:48.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:01:48.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:01:48.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:01:48.157 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:01:48.157 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:48.157 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:48.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T16:01:48.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:01:48.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:01:48.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-17T16:01:48.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:01:48.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:01:48.185 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:01:48.185 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:48.185 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:48.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-mon.a.asok 2024-09-17T16:01:48.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-17T16:01:48.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-mon.a.asok log flush 2024-09-17T16:01:48.256 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:01:48.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse4 2024-09-17T16:01:48.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-17T16:01:48.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:01:48.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:01:48.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:01:48.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-17T16:01:48.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-17T16:01:48.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:01:48.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:01:48.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:01:48.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-17T16:01:48.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-17T16:01:48.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:01:48.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:01:48.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-17T16:01:48.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:01:48.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:01:48.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:01:48.268 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:01:48.268 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:01:48.268 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:01:48.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T16:01:48.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' 2024-09-17T16:01:48.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:01:48.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:01:48.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:01:48.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-17T16:01:48.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-17T16:01:48.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:01:48.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:01:48.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:01:48.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:01:48.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:01:48.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse4 2024-09-17T16:01:48.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-17T16:01:48.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:01:48.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a0cfd6c2-8383-480b-9ccf-f6f10c1edb6e 2024-09-17T16:01:48.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a0cfd6c2-8383-480b-9ccf-f6f10c1edb6e' 2024-09-17T16:01:48.270 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 a0cfd6c2-8383-480b-9ccf-f6f10c1edb6e 2024-09-17T16:01:48.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:01:48.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDsp+lmq63OEBAAb2dCE6EdJTeugSwWGGCP/A== 2024-09-17T16:01:48.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDsp+lmq63OEBAAb2dCE6EdJTeugSwWGGCP/A=="}' 2024-09-17T16:01:48.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a0cfd6c2-8383-480b-9ccf-f6f10c1edb6e -i td/test-erasure-code-plugins/0/new.json 2024-09-17T16:01:48.535 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:01:48.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-17T16:01:48.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 --mkfs --key AQDsp+lmq63OEBAAb2dCE6EdJTeugSwWGGCP/A== --osd-uuid a0cfd6c2-8383-480b-9ccf-f6f10c1edb6e 2024-09-17T16:01:48.573 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:48.571+0000 7f7bfe016640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:01:48.603 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:48.601+0000 7f7bfe016640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:01:48.604 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:48.603+0000 7f7bfe016640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:01:48.605 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:48.603+0000 7f7bfe016640 -1 bdev(0x56409f65dc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:01:48.605 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:48.603+0000 7f7bfe016640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-17T16:01:53.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-17T16:01:53.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:01:53.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:01:53.841 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:01:53.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:01:54.379 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:01:54.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:01:54.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f29b6048-ffe9-430b-9f47-a9d4b8d8328b --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 2024-09-17T16:01:54.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:01:54.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:01:54.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:01:54.405 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:54.403+0000 7fc516e2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:01:54.414 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:54.412+0000 7fc516e2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:01:54.415 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:54.414+0000 7fc516e2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:01:54.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:01:54.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:01:54.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:01:54.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:01:54.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:01:54.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:54.829 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:01:54.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:01:54.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:54.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:55.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:55.760 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:55.758+0000 7fc516e2e640 -1 Falling back to public interface 2024-09-17T16:01:56.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:56.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:56.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:01:56.256 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:01:56.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:56.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:56.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:57.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:57.681 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:01:57.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:57.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:01:57.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:57.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:58.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:01:58.900 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:01:58.898+0000 7fc516e2e640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:01:59.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:01:59.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:01:59.075 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:01:59.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:01:59.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:01:59.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:01:59.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:00.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:00.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:00.513 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:02:00.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:02:00.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:00.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:01.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:02.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:02.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:02.009 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:02:02.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:02:02.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:02.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:02.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:03.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:03.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:03.431 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:02:03.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:02:03.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:03.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:03.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:04.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:04.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:04.960 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:02:04.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:02:04.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:04.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:05.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:06.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:06.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:06.378 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:02:06.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:02:06.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:06.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:06.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:07.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:07.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:07.800 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:02:07.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:02:07.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:07.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:08.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:09.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:09.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:09.215 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:02:09.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:02:09.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:09.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:09.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:10.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:10.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:10.693 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:02:10.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:02:10.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:10.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:11.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:12.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:12.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:12.491 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:02:12.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:02:12.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:12.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:13.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:14.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:14.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:14.671 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:02:14.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:02:14.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:14.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:15.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:16.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:16.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:16.613 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:02:16.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:02:16.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:16.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:17.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:18.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:18.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:18.735 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:02:18.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:02:18.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:18.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:19.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:20.787 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:02:20.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:02:20.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:20.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:21.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:22.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:22.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:22.314 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:02:22.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:02:22.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:22.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:22.752 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/445159630,v1:127.0.0.1:6803/445159630] [v2:127.0.0.1:6804/445159630,v1:127.0.0.1:6805/445159630] exists,up a0cfd6c2-8383-480b-9ccf-f6f10c1edb6e 2024-09-17T16:02:22.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:02:22.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:02:22.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:02:22.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-17T16:02:22.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:02:22.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:02:22.754 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:02:22.754 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:02:22.754 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:02:22.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27518/ceph-osd.0.asok 2024-09-17T16:02:22.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-17T16:02:22.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27518/ceph-osd.0.asok log flush 2024-09-17T16:02:22.832 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:02:22.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-17T16:02:22.843 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:01:46.977+0000 7f0649437c80 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-17T16:02:22.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/osd.0.log 2024-09-17T16:02:22.844 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:01:55.759+0000 7fc516e2e640 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-17T16:02:22.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-17T16:02:22.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T16:02:22.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:02:22.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T16:02:22.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:02:22.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:02:22.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:02:22.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:02:22.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:02:22.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:02:22.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:02:22.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:02:22.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:02:22.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:02:22.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:02:22.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:02:22.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:02:22.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:02:22.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:02:22.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:02:22.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:02:22.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:02:22.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T16:02:22.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:02:22.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:02:22.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:02:22.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T16:02:22.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:02:22.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:02:22.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:54: TEST_preload_warning: return 0 2024-09-17T16:02:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-17T16:02:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/test-erasure-code-plugins 0 2024-09-17T16:02:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-17T16:02:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-17T16:02:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-17T16:02:22.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:02:22.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:02:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:02:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:02:22.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:02:22.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:02:22.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:02:22.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:02:22.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:02:22.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:02:22.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:02:22.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:02:22.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:02:22.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:02:22.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:02:22.968 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:02:22.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:02:22.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-17T16:02:22.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-17T16:02:22.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:02:22.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:02:22.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27518 2024-09-17T16:02:22.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27518 2024-09-17T16:02:22.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:02:22.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:02:22.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-17T16:02:22.973 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T16:02:22.973 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T16:02:23.000 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code.sh... 2024-09-17T16:02:23.000 DEBUG:teuthology.orchestra.run.smithi162:workunit test erasure-code/test-erasure-code.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh 2024-09-17T16:02:23.061 INFO:tasks.workunit.client.0.smithi162.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-17T16:02:23.068 INFO:tasks.workunit.client.0.smithi162.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-17T16:02:23.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T16:02:23.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T16:02:23.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-17T16:02:23.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-17T16:02:23.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-17T16:02:23.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-17T16:02:23.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-17T16:02:23.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-17T16:02:23.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/test-erasure-code 2024-09-17T16:02:23.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:22: run: local dir=td/test-erasure-code 2024-09-17T16:02:23.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:23: run: shift 2024-09-17T16:02:23.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: export CEPH_MON=127.0.0.1:7101 2024-09-17T16:02:23.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: CEPH_MON=127.0.0.1:7101 2024-09-17T16:02:23.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:26: run: export CEPH_ARGS 2024-09-17T16:02:23.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: uuidgen 2024-09-17T16:02:23.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: CEPH_ARGS+='--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none ' 2024-09-17T16:02:23.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:02:23.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:30: run: setup td/test-erasure-code 2024-09-17T16:02:23.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code 2024-09-17T16:02:23.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code 2024-09-17T16:02:23.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code 2024-09-17T16:02:23.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:02:23.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code KILL 2024-09-17T16:02:23.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:02:23.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:02:23.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:02:23.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:02:23.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:02:23.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:02:23.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:02:23.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:02:23.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:02:23.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:02:23.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:02:23.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:02:23.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:02:23.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:02:23.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:02:23.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:02:23.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:02:23.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:02:23.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code 2024-09-17T16:02:23.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:02:23.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:02:23.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:02:23.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56062 2024-09-17T16:02:23.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:02:23.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:02:23.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code 2024-09-17T16:02:23.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:02:23.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:02:23.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:02:23.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56062 2024-09-17T16:02:23.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:02:23.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-17T16:02:23.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-17T16:02:23.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:02:23.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code 1' TERM HUP INT 2024-09-17T16:02:23.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:31: run: run_mon td/test-erasure-code a 2024-09-17T16:02:23.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code 2024-09-17T16:02:23.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:02:23.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:02:23.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:02:23.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code/a 2024-09-17T16:02:23.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code/a --run-dir=td/test-erasure-code 2024-09-17T16:02:23.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:02:23.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:02:23.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:02:23.523 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:02:23.523 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:02:23.523 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:02:23.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:02:23.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code/a '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code/log --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T16:02:23.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:02:23.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:02:23.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:02:23.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:02:23.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:02:23.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:02:23.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:02:23.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:02:23.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:02:23.718 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:02:23.718 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:02:23.718 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:02:23.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56062/ceph-mon.a.asok 2024-09-17T16:02:23.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:02:23.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56062/ceph-mon.a.asok config get fsid 2024-09-17T16:02:23.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:02:23.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:02:23.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:02:23.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:02:23.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:02:23.800 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:02:23.800 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:02:23.800 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:02:23.801 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:02:23.801 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:02:23.801 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:02:23.801 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56062/ceph-mon.a.asok 2024-09-17T16:02:23.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:02:23.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56062/ceph-mon.a.asok config get mon_host 2024-09-17T16:02:23.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:32: run: run_mgr td/test-erasure-code x 2024-09-17T16:02:23.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code 2024-09-17T16:02:23.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:02:23.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:02:23.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:02:23.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code/x 2024-09-17T16:02:23.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:02:24.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:02:24.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:02:24.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:02:24.722 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:02:24.722 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:02:24.722 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:02:24.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:02:24.723 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:02:24.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code/x '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:02:24.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: get_asok_path mon.a 2024-09-17T16:02:24.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:02:24.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:02:24.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:02:24.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:02:24.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:02:24.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56062/ceph-mon.a.asok 2024-09-17T16:02:24.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: CEPH_ARGS= 2024-09-17T16:02:24.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: ceph --admin-daemon /tmp/ceph-asok.56062/ceph-mon.a.asok log flush 2024-09-17T16:02:24.823 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:02:24.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:35: run: grep 'load: jerasure.*lrc' td/test-erasure-code/mon.a.log 2024-09-17T16:02:24.832 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:02:23.555+0000 7fd893f1dc80 0 load: jerasure load: lrc 2024-09-17T16:02:24.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: seq 0 10 2024-09-17T16:02:24.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-17T16:02:24.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 0 2024-09-17T16:02:24.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-17T16:02:24.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:02:24.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:02:24.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:02:24.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/0 2024-09-17T16:02:24.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:02:24.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:02:24.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:02:24.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:02:24.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/0' 2024-09-17T16:02:24.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/0/journal' 2024-09-17T16:02:24.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:02:24.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:02:24.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-17T16:02:24.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:02:24.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:02:24.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:02:24.838 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:02:24.838 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:02:24.838 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:02:24.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:02:24.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:02:24.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:02:24.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:02:24.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:02:24.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-17T16:02:24.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-17T16:02:24.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:02:24.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:02:24.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:02:24.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:02:24.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:02:24.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:02:24.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/0 2024-09-17T16:02:24.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:02:24.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8d7ca4b1-c574-48d8-9b6e-594957c7d01a 2024-09-17T16:02:24.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8d7ca4b1-c574-48d8-9b6e-594957c7d01a' 2024-09-17T16:02:24.841 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 8d7ca4b1-c574-48d8-9b6e-594957c7d01a 2024-09-17T16:02:24.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:02:24.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAQqOlm98fRMhAARnbe26UJxionUKJKjIM92g== 2024-09-17T16:02:24.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAQqOlm98fRMhAARnbe26UJxionUKJKjIM92g=="}' 2024-09-17T16:02:24.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8d7ca4b1-c574-48d8-9b6e-594957c7d01a -i td/test-erasure-code/0/new.json 2024-09-17T16:02:25.116 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:02:25.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/0/new.json 2024-09-17T16:02:25.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQqOlm98fRMhAARnbe26UJxionUKJKjIM92g== --osd-uuid 8d7ca4b1-c574-48d8-9b6e-594957c7d01a 2024-09-17T16:02:25.156 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:02:25.153+0000 7fe473617640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:02:25.185 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:02:25.182+0000 7fe473617640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:02:25.186 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:02:25.183+0000 7fe473617640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:02:25.187 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:02:25.184+0000 7fe473617640 -1 bdev(0x55d308e25c00 td/test-erasure-code/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:02:25.187 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:02:25.184+0000 7fe473617640 -1 bluestore(td/test-erasure-code/0) _read_fsid unparsable uuid 2024-09-17T16:02:30.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/0/keyring 2024-09-17T16:02:30.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:02:30.925 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:02:30.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:02:30.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:02:31.464 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:02:31.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:02:31.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:02:31.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:02:31.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:02:31.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:02:31.491 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:02:31.489+0000 7f77937a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:02:31.500 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:02:31.498+0000 7f77937a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:02:31.501 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:02:31.499+0000 7f77937a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:02:31.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:02:31.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:02:31.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:02:31.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:02:31.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:02:31.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:31.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:02:31.916 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:02:31.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:31.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:32.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:32.600 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:02:32.598+0000 7f77937a8640 -1 Falling back to public interface 2024-09-17T16:02:33.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:33.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:33.344 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:02:33.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:02:33.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:33.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:33.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:34.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:34.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:34.781 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:02:34.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:02:34.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:34.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:35.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:35.371 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:02:35.369+0000 7f77937a8640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:02:36.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:36.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:36.169 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:02:36.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:02:36.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:36.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:36.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:37.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:37.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:37.560 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:02:37.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:02:37.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:37.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:37.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:38.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:38.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:38.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:02:38.974 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:02:38.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:38.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:39.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:40.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:40.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:40.374 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:02:40.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:02:40.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:40.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:40.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:41.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:41.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:41.767 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:02:41.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:02:41.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:41.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:42.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:43.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:43.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:43.206 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:02:43.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:02:43.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:43.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:43.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:44.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:44.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:44.639 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:02:44.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:02:44.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:44.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:46.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:46.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:46.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:02:46.195 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:02:46.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:46.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:46.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:47.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:47.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:47.596 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:02:47.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:02:47.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:47.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:48.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:49.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:49.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:49.189 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:02:49.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:02:49.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:49.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:50.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:51.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:51.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:51.067 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:02:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:02:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:52.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:53.176 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:02:53.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:02:53.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:53.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:53.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:54.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:54.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:54.798 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:02:54.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:02:54.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:54.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:55.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:56.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:56.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:56.206 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:02:56.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:02:56.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:56.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:56.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:02:57.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:02:57.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:02:57.909 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:02:57.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:02:57.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:02:57.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:02:58.620 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3900008474,v1:127.0.0.1:6803/3900008474] [v2:127.0.0.1:6804/3900008474,v1:127.0.0.1:6805/3900008474] exists,up 8d7ca4b1-c574-48d8-9b6e-594957c7d01a 2024-09-17T16:02:58.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:02:58.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:02:58.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:02:58.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-17T16:02:58.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 1 2024-09-17T16:02:58.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-17T16:02:58.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:02:58.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T16:02:58.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:02:58.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/1 2024-09-17T16:02:58.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:02:58.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:02:58.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:02:58.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:02:58.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/1' 2024-09-17T16:02:58.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/1/journal' 2024-09-17T16:02:58.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:02:58.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:02:58.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-17T16:02:58.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:02:58.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:02:58.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:02:58.626 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:02:58.626 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:02:58.626 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:02:58.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:02:58.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:02:58.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:02:58.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:02:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:02:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-17T16:02:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-17T16:02:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:02:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:02:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:02:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:02:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:02:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:02:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/1 2024-09-17T16:02:58.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:02:58.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e7ade82a-f1b0-4541-94dc-8bc680bbef5d 2024-09-17T16:02:58.628 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 e7ade82a-f1b0-4541-94dc-8bc680bbef5d 2024-09-17T16:02:58.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e7ade82a-f1b0-4541-94dc-8bc680bbef5d' 2024-09-17T16:02:58.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:02:58.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAyqOlmyuo4JhAAjsK8fLb/MkecDIOgS4iR+g== 2024-09-17T16:02:58.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAyqOlmyuo4JhAAjsK8fLb/MkecDIOgS4iR+g=="}' 2024-09-17T16:02:58.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e7ade82a-f1b0-4541-94dc-8bc680bbef5d -i td/test-erasure-code/1/new.json 2024-09-17T16:02:59.089 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:02:59.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/1/new.json 2024-09-17T16:02:59.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAyqOlmyuo4JhAAjsK8fLb/MkecDIOgS4iR+g== --osd-uuid e7ade82a-f1b0-4541-94dc-8bc680bbef5d 2024-09-17T16:02:59.131 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:02:59.129+0000 7f13f6a2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:02:59.138 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:02:59.136+0000 7f13f6a2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:02:59.139 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:02:59.137+0000 7f13f6a2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:02:59.139 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:02:59.137+0000 7f13f6a2e640 -1 bdev(0x5649f8a1bc00 td/test-erasure-code/1/block) open stat got: (1) Operation not permitted 2024-09-17T16:02:59.139 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:02:59.138+0000 7f13f6a2e640 -1 bluestore(td/test-erasure-code/1) _read_fsid unparsable uuid 2024-09-17T16:03:05.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/1/keyring 2024-09-17T16:03:05.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:03:05.127 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T16:03:05.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T16:03:05.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:03:05.663 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T16:03:05.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T16:03:05.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:03:05.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:03:05.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:03:05.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:03:05.689 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:03:05.687+0000 7fba21083640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:03:05.701 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:03:05.699+0000 7fba21083640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:03:05.702 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:03:05.700+0000 7fba21083640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:03:06.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T16:03:06.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:03:06.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:03:06.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:03:06.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:03:06.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:06.116 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:03:06.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:03:06.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:06.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:06.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:07.305 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:03:07.302+0000 7fba21083640 -1 Falling back to public interface 2024-09-17T16:03:07.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:07.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:07.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:03:07.534 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:03:07.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:07.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:07.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:08.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:08.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:08.969 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:03:08.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:03:08.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:08.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:09.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:09.996 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:03:09.993+0000 7fba21083640 -1 osd.1 0 log_to_monitors true 2024-09-17T16:03:10.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:10.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:10.403 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:03:10.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:03:10.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:10.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:10.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:11.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:11.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:11.809 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:03:11.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:03:11.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:11.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:12.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:13.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:13.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:13.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:03:13.240 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:03:13.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:13.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:13.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:14.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:14.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:14.644 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:03:14.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:03:14.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:14.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:15.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:16.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:16.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:16.243 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:03:16.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:03:16.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:16.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:16.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:17.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:17.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:17.641 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:03:17.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:03:17.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:17.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:18.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:19.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:19.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:19.148 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:03:19.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:03:19.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:19.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:19.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:20.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:20.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:20.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:03:20.573 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:03:20.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:20.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:20.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:21.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:21.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:21.980 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:03:21.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:03:21.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:21.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:22.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:23.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:23.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:23.672 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:03:23.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:03:23.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:23.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:24.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:25.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:25.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:25.412 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:03:25.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:03:25.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:25.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:26.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:27.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:27.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:27.241 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:03:27.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:03:27.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:27.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:28.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:29.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:29.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:29.020 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:03:29.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:03:29.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:29.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:29.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:30.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:30.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:30.922 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:03:30.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:03:30.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:30.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:31.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:32.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:32.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:32.446 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:03:32.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:03:32.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:32.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:33.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:34.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:34.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:34.076 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:03:34.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:03:34.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:34.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:03:34.501 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3012409212,v1:127.0.0.1:6811/3012409212] [v2:127.0.0.1:6812/3012409212,v1:127.0.0.1:6813/3012409212] exists,up e7ade82a-f1b0-4541-94dc-8bc680bbef5d 2024-09-17T16:03:34.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:03:34.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:03:34.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:03:34.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-17T16:03:34.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 2 2024-09-17T16:03:34.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-17T16:03:34.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:03:34.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T16:03:34.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:03:34.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/2 2024-09-17T16:03:34.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:03:34.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:03:34.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:03:34.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:03:34.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/2' 2024-09-17T16:03:34.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/2/journal' 2024-09-17T16:03:34.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:03:34.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:03:34.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-17T16:03:34.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:03:34.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:03:34.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:03:34.507 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:03:34.507 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:03:34.507 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:03:34.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:03:34.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:03:34.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:03:34.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:03:34.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:03:34.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-17T16:03:34.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-17T16:03:34.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:03:34.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:03:34.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:03:34.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:03:34.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:03:34.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:03:34.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/2 2024-09-17T16:03:34.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:03:34.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=db542414-758a-4f74-ba7a-6ebb3e16a00c 2024-09-17T16:03:34.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 db542414-758a-4f74-ba7a-6ebb3e16a00c' 2024-09-17T16:03:34.509 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 db542414-758a-4f74-ba7a-6ebb3e16a00c 2024-09-17T16:03:34.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:03:34.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBWqOlm9rUOHxAAYj9F5wDNZXJLfBxnHNYMoQ== 2024-09-17T16:03:34.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBWqOlm9rUOHxAAYj9F5wDNZXJLfBxnHNYMoQ=="}' 2024-09-17T16:03:34.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new db542414-758a-4f74-ba7a-6ebb3e16a00c -i td/test-erasure-code/2/new.json 2024-09-17T16:03:35.003 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:03:35.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/2/new.json 2024-09-17T16:03:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBWqOlm9rUOHxAAYj9F5wDNZXJLfBxnHNYMoQ== --osd-uuid db542414-758a-4f74-ba7a-6ebb3e16a00c 2024-09-17T16:03:35.044 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:03:35.042+0000 7fca09fae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:03:35.067 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:03:35.066+0000 7fca09fae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:03:35.068 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:03:35.067+0000 7fca09fae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:03:35.069 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:03:35.067+0000 7fca09fae640 -1 bdev(0x55d7283d5c00 td/test-erasure-code/2/block) open stat got: (1) Operation not permitted 2024-09-17T16:03:35.069 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:03:35.067+0000 7fca09fae640 -1 bluestore(td/test-erasure-code/2) _read_fsid unparsable uuid 2024-09-17T16:03:41.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/2/keyring 2024-09-17T16:03:41.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:03:41.225 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T16:03:41.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T16:03:41.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:03:41.767 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T16:03:41.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T16:03:41.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:03:41.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:03:41.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:03:41.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:03:41.794 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:03:41.792+0000 7f66e32b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:03:41.805 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:03:41.803+0000 7f66e32b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:03:41.806 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:03:41.805+0000 7f66e32b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:03:42.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T16:03:42.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:03:42.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:03:42.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:03:42.219 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:03:42.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:03:42.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:42.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:03:42.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:42.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:03:42.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:43.401 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:03:43.399+0000 7f66e32b6640 -1 Falling back to public interface 2024-09-17T16:03:43.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:43.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:43.645 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:03:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:03:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:03:44.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:45.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:45.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:45.081 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:03:45.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:03:45.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:45.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:03:45.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:46.285 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:03:46.283+0000 7f66e32b6640 -1 osd.2 0 log_to_monitors true 2024-09-17T16:03:46.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:46.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:46.516 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:03:46.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:03:46.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:46.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:03:47.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:48.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:48.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:48.110 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:03:48.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:03:48.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:48.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:03:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:49.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:49.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:49.510 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:03:49.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:03:49.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:49.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:03:49.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:50.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:50.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:50.935 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:03:50.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:03:50.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:50.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:03:51.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:52.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:52.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:52.348 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:03:52.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:03:52.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:52.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:03:52.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:53.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:53.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:53.747 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:03:53.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:03:53.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:53.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:03:54.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:55.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:55.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:55.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:03:55.291 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:03:55.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:55.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:03:55.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:56.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:56.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:56.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:03:56.715 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:03:56.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:56.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:03:57.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:58.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:58.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:58.212 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:03:58.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:03:58.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:58.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:03:58.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:03:59.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:03:59.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:03:59.610 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:03:59.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:03:59.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:03:59.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:04:00.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:01.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:01.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:01.427 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:04:01.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:04:01.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:01.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:04:02.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:03.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:03.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:03.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:04:03.616 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:04:03.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:03.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:04:04.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:05.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:05.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:05.384 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:04:05.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:04:05.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:05.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:04:06.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:07.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:07.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:07.554 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:04:07.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:04:07.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:07.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:04:08.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:09.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:09.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:09.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:04:09.606 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:04:09.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:09.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:04:10.035 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1462601604,v1:127.0.0.1:6819/1462601604] [v2:127.0.0.1:6820/1462601604,v1:127.0.0.1:6821/1462601604] exists,up db542414-758a-4f74-ba7a-6ebb3e16a00c 2024-09-17T16:04:10.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:04:10.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:04:10.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:04:10.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-17T16:04:10.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 3 2024-09-17T16:04:10.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-17T16:04:10.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:04:10.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T16:04:10.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:04:10.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/3 2024-09-17T16:04:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:04:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:04:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:04:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:04:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-09-17T16:04:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-09-17T16:04:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:04:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:04:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-17T16:04:10.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:04:10.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:04:10.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:04:10.041 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:04:10.041 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:04:10.041 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:04:10.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:04:10.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:04:10.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:04:10.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:04:10.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:04:10.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-17T16:04:10.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-17T16:04:10.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:04:10.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:04:10.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:04:10.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:04:10.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:04:10.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:04:10.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/3 2024-09-17T16:04:10.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:04:10.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a1b58765-3583-41df-91a6-d67cf3dd9c17 2024-09-17T16:04:10.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 a1b58765-3583-41df-91a6-d67cf3dd9c17' 2024-09-17T16:04:10.042 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 a1b58765-3583-41df-91a6-d67cf3dd9c17 2024-09-17T16:04:10.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:04:10.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB6qOlm6pJUAxAAYTaCxvEAyX5S4JcM5WdbVA== 2024-09-17T16:04:10.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB6qOlm6pJUAxAAYTaCxvEAyX5S4JcM5WdbVA=="}' 2024-09-17T16:04:10.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a1b58765-3583-41df-91a6-d67cf3dd9c17 -i td/test-erasure-code/3/new.json 2024-09-17T16:04:10.553 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:04:10.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/3/new.json 2024-09-17T16:04:10.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6qOlm6pJUAxAAYTaCxvEAyX5S4JcM5WdbVA== --osd-uuid a1b58765-3583-41df-91a6-d67cf3dd9c17 2024-09-17T16:04:10.593 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:10.591+0000 7f5515989640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:04:10.596 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:10.594+0000 7f5515989640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:04:10.597 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:10.596+0000 7f5515989640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:04:10.598 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:10.596+0000 7f5515989640 -1 bdev(0x55db1d3f5c00 td/test-erasure-code/3/block) open stat got: (1) Operation not permitted 2024-09-17T16:04:10.598 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:10.596+0000 7f5515989640 -1 bluestore(td/test-erasure-code/3) _read_fsid unparsable uuid 2024-09-17T16:04:16.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/3/keyring 2024-09-17T16:04:16.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:04:16.715 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-17T16:04:16.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T16:04:16.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:04:17.280 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T16:04:17.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T16:04:17.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:04:17.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:04:17.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:04:17.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:04:17.307 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:17.305+0000 7f444f028640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:04:17.323 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:17.322+0000 7f444f028640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:04:17.325 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:17.323+0000 7f444f028640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:04:17.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T16:04:17.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:04:17.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:04:17.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:04:17.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:04:17.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:17.738 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:04:17.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:04:17.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:17.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:18.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:18.429 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:18.427+0000 7f444f028640 -1 Falling back to public interface 2024-09-17T16:04:19.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:19.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:19.165 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:04:19.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:04:19.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:19.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:19.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:20.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:20.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:20.608 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:04:20.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:04:20.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:20.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:21.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:21.602 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:21.600+0000 7f444f028640 -1 osd.3 0 log_to_monitors true 2024-09-17T16:04:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:04:22.016 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:04:22.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:22.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:22.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:23.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:23.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:23.540 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:04:23.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:04:23.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:23.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:23.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:24.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:24.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:24.937 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:04:24.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:04:24.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:24.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:25.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:26.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:26.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:26.376 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:04:26.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:04:26.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:26.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:26.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:27.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:27.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:27.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:04:27.793 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:04:27.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:27.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:28.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:29.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:29.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:29.221 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:04:29.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:04:29.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:29.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:29.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:30.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:30.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:30.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:04:30.662 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:04:30.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:30.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:31.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:32.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:32.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:32.057 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:04:32.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:04:32.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:32.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:32.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:33.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:33.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:33.498 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:04:33.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:04:33.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:33.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:34.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:35.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:35.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:35.164 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:04:35.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:04:35.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:35.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:35.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:36.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:36.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:36.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:04:36.955 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:04:36.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:36.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:37.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:38.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:38.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:38.656 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:04:38.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:04:38.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:38.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:39.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:40.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:40.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:40.284 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:04:40.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:04:40.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:40.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:41.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:42.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:42.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:42.112 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:04:42.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:04:42.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:42.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:42.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:43.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:43.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:43.873 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:04:43.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:04:43.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:43.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:44.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:45.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:45.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:45.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:04:45.634 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:04:45.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:45.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:04:46.065 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1472935139,v1:127.0.0.1:6827/1472935139] [v2:127.0.0.1:6828/1472935139,v1:127.0.0.1:6829/1472935139] exists,up a1b58765-3583-41df-91a6-d67cf3dd9c17 2024-09-17T16:04:46.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:04:46.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:04:46.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:04:46.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-17T16:04:46.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 4 2024-09-17T16:04:46.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-17T16:04:46.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:04:46.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-17T16:04:46.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:04:46.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/4 2024-09-17T16:04:46.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:04:46.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:04:46.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:04:46.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:04:46.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/4' 2024-09-17T16:04:46.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/4/journal' 2024-09-17T16:04:46.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:04:46.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:04:46.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-17T16:04:46.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:04:46.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:04:46.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:04:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:04:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:04:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:04:46.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:04:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:04:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:04:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:04:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:04:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-17T16:04:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-17T16:04:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:04:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:04:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:04:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:04:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:04:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:04:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/4 2024-09-17T16:04:46.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:04:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1f08568f-d989-4a1e-bcec-334d67886670 2024-09-17T16:04:46.072 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 1f08568f-d989-4a1e-bcec-334d67886670 2024-09-17T16:04:46.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 1f08568f-d989-4a1e-bcec-334d67886670' 2024-09-17T16:04:46.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:04:46.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCeqOlmAMMWBRAA5Mgqk1QxbI+2Oksb5j2Vxw== 2024-09-17T16:04:46.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCeqOlmAMMWBRAA5Mgqk1QxbI+2Oksb5j2Vxw=="}' 2024-09-17T16:04:46.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1f08568f-d989-4a1e-bcec-334d67886670 -i td/test-erasure-code/4/new.json 2024-09-17T16:04:46.532 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:04:46.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/4/new.json 2024-09-17T16:04:46.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCeqOlmAMMWBRAA5Mgqk1QxbI+2Oksb5j2Vxw== --osd-uuid 1f08568f-d989-4a1e-bcec-334d67886670 2024-09-17T16:04:46.573 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:46.570+0000 7f4980432640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:04:46.581 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:46.578+0000 7f4980432640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:04:46.582 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:46.579+0000 7f4980432640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:04:46.583 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:46.580+0000 7f4980432640 -1 bdev(0x55f9c4dfdc00 td/test-erasure-code/4/block) open stat got: (1) Operation not permitted 2024-09-17T16:04:46.583 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:46.580+0000 7f4980432640 -1 bluestore(td/test-erasure-code/4) _read_fsid unparsable uuid 2024-09-17T16:04:52.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/4/keyring 2024-09-17T16:04:52.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:04:52.553 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-17T16:04:52.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-17T16:04:52.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:04:53.123 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-17T16:04:53.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-17T16:04:53.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:04:53.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:04:53.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:04:53.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:04:53.149 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:53.147+0000 7febe8822640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:04:53.151 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:53.149+0000 7febe8822640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:04:53.152 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:53.150+0000 7febe8822640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:04:53.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-17T16:04:53.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:04:53.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-17T16:04:53.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:04:53.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:04:53.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:53.576 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:04:53.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:04:53.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:53.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:04:53.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:54.009 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:54.007+0000 7febe8822640 -1 Falling back to public interface 2024-09-17T16:04:54.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:54.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:54.998 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:04:54.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:04:54.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:54.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:04:55.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:56.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:56.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:56.429 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:04:56.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:04:56.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:56.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:04:56.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:56.838 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:04:56.837+0000 7febe8822640 -1 osd.4 0 log_to_monitors true 2024-09-17T16:04:57.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:57.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:57.821 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:04:57.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:04:57.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:57.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:04:58.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:04:59.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:04:59.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:04:59.290 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:04:59.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:04:59.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:04:59.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:04:59.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:00.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:00.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:00.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:05:00.736 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:05:00.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:00.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:05:01.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:02.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:02.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:02.201 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:05:02.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:05:02.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:02.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:05:02.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:03.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:03.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:03.697 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:05:03.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:05:03.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:03.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:05:04.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:05.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:05.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:05.113 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:05:05.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:05:05.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:05.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:05:05.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:06.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:06.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:06.569 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:05:06.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:05:06.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:06.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:05:07.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:08.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:08.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:08.022 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:05:08.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:05:08.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:08.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:05:08.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:09.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:09.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:09.448 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:05:09.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:05:09.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:09.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:05:10.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:11.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:11.197 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:05:11.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:11.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:05:11.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:11.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:05:12.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:13.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:13.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:13.066 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:05:13.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:05:13.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:13.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:05:14.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:15.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:15.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:15.467 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:05:15.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:05:15.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:15.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:05:16.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:17.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:17.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:17.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:05:17.189 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:05:17.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:17.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:05:17.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:18.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:18.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:18.897 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:05:18.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:05:18.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:18.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:05:19.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:20.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:20.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:20.561 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:05:20.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:05:20.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:20.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:05:21.018 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2942312339,v1:127.0.0.1:6835/2942312339] [v2:127.0.0.1:6836/2942312339,v1:127.0.0.1:6837/2942312339] exists,up 1f08568f-d989-4a1e-bcec-334d67886670 2024-09-17T16:05:21.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:05:21.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:05:21.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:05:21.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-17T16:05:21.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 5 2024-09-17T16:05:21.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-17T16:05:21.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:05:21.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-17T16:05:21.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:05:21.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/5 2024-09-17T16:05:21.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:05:21.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:05:21.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:05:21.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:05:21.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-09-17T16:05:21.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-09-17T16:05:21.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:05:21.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:05:21.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-17T16:05:21.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:05:21.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:05:21.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:05:21.024 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:05:21.024 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:05:21.024 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:05:21.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:05:21.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:05:21.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:05:21.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:05:21.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:05:21.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-17T16:05:21.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-17T16:05:21.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:05:21.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:05:21.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:05:21.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:05:21.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:05:21.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:05:21.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/5 2024-09-17T16:05:21.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:05:21.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=316d12e1-735e-43da-ba4d-68452198e455 2024-09-17T16:05:21.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 316d12e1-735e-43da-ba4d-68452198e455' 2024-09-17T16:05:21.025 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 316d12e1-735e-43da-ba4d-68452198e455 2024-09-17T16:05:21.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:05:21.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDBqOlm80lBAhAAkihj0pZrU+ZeuOt3QXL78g== 2024-09-17T16:05:21.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDBqOlm80lBAhAAkihj0pZrU+ZeuOt3QXL78g=="}' 2024-09-17T16:05:21.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 316d12e1-735e-43da-ba4d-68452198e455 -i td/test-erasure-code/5/new.json 2024-09-17T16:05:21.515 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:05:21.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/5/new.json 2024-09-17T16:05:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDBqOlm80lBAhAAkihj0pZrU+ZeuOt3QXL78g== --osd-uuid 316d12e1-735e-43da-ba4d-68452198e455 2024-09-17T16:05:21.556 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:05:21.554+0000 7f83aa617640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:05:21.566 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:05:21.564+0000 7f83aa617640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:05:21.567 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:05:21.565+0000 7f83aa617640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:05:21.567 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:05:21.565+0000 7f83aa617640 -1 bdev(0x5585c9ddbc00 td/test-erasure-code/5/block) open stat got: (1) Operation not permitted 2024-09-17T16:05:21.567 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:05:21.566+0000 7f83aa617640 -1 bluestore(td/test-erasure-code/5) _read_fsid unparsable uuid 2024-09-17T16:05:28.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/5/keyring 2024-09-17T16:05:28.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:05:28.051 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2024-09-17T16:05:28.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-17T16:05:28.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:05:28.585 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2024-09-17T16:05:28.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-17T16:05:28.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:05:28.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:05:28.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:05:28.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:05:28.612 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:05:28.610+0000 7f3767aa5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:05:28.624 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:05:28.622+0000 7f3767aa5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:05:28.625 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:05:28.624+0000 7f3767aa5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:05:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-17T16:05:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:05:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T16:05:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:05:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:05:29.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:29.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:05:29.006 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:05:29.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:29.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:29.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:29.989 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:05:29.986+0000 7f3767aa5640 -1 Falling back to public interface 2024-09-17T16:05:30.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:30.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:30.416 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:05:30.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:05:30.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:30.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:30.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:31.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:31.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:31.865 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:05:31.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:05:31.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:31.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:32.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:33.053 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:05:33.050+0000 7f3767aa5640 -1 osd.5 0 log_to_monitors true 2024-09-17T16:05:33.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:33.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:33.302 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:05:33.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:05:33.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:33.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:33.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:34.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:34.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:34.762 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:05:34.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:05:34.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:34.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:35.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:36.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:36.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:36.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:05:36.197 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:05:36.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:36.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:36.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:37.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:37.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:37.630 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:05:37.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:05:37.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:37.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:38.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:39.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:39.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:39.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:05:39.031 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:05:39.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:39.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:39.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:40.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:40.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:40.493 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:05:40.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:05:40.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:40.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:41.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:42.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:42.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:42.002 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:05:42.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:05:42.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:42.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:42.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:43.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:43.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:43.406 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:05:43.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:05:43.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:43.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:43.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:44.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:44.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:44.873 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:05:44.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:05:44.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:44.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:45.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:46.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:46.518 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:05:46.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:46.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:05:46.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:46.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:47.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:48.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:48.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:48.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:05:48.194 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:05:48.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:48.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:49.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:50.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:50.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:50.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:05:50.050 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:05:50.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:50.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:50.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:51.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:51.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:51.862 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:05:51.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:05:51.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:51.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:52.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:53.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:53.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:53.685 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:05:53.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:05:53.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:53.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:54.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:55.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:55.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:55.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:05:55.821 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:05:55.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:55.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:56.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:05:57.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:05:57.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:05:57.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:05:57.286 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:05:57.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:05:57.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:05:57.712 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1181673015,v1:127.0.0.1:6843/1181673015] [v2:127.0.0.1:6844/1181673015,v1:127.0.0.1:6845/1181673015] exists,up 316d12e1-735e-43da-ba4d-68452198e455 2024-09-17T16:05:57.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:05:57.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:05:57.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:05:57.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-17T16:05:57.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 6 2024-09-17T16:05:57.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-17T16:05:57.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:05:57.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-17T16:05:57.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:05:57.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/6 2024-09-17T16:05:57.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:05:57.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:05:57.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:05:57.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:05:57.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-09-17T16:05:57.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-09-17T16:05:57.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:05:57.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:05:57.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-17T16:05:57.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:05:57.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:05:57.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:05:57.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:05:57.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:05:57.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:05:57.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:05:57.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:05:57.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:05:57.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:05:57.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:05:57.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-17T16:05:57.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-17T16:05:57.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:05:57.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:05:57.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:05:57.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:05:57.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:05:57.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:05:57.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/6 2024-09-17T16:05:57.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:05:57.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=847dfb63-8a98-445f-976b-8f106467502e 2024-09-17T16:05:57.720 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 847dfb63-8a98-445f-976b-8f106467502e 2024-09-17T16:05:57.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 847dfb63-8a98-445f-976b-8f106467502e' 2024-09-17T16:05:57.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:05:57.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDlqOlm0XeuKxAAZgqTaVZX5e6qu51yukvIzg== 2024-09-17T16:05:57.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDlqOlm0XeuKxAAZgqTaVZX5e6qu51yukvIzg=="}' 2024-09-17T16:05:57.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 847dfb63-8a98-445f-976b-8f106467502e -i td/test-erasure-code/6/new.json 2024-09-17T16:05:58.244 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:05:58.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/6/new.json 2024-09-17T16:05:58.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDlqOlm0XeuKxAAZgqTaVZX5e6qu51yukvIzg== --osd-uuid 847dfb63-8a98-445f-976b-8f106467502e 2024-09-17T16:05:58.285 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:05:58.283+0000 7f88e0224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:05:58.287 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:05:58.285+0000 7f88e0224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:05:58.288 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:05:58.286+0000 7f88e0224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:05:58.288 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:05:58.287+0000 7f88e0224640 -1 bdev(0x55e12b6edc00 td/test-erasure-code/6/block) open stat got: (1) Operation not permitted 2024-09-17T16:05:58.289 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:05:58.287+0000 7f88e0224640 -1 bluestore(td/test-erasure-code/6) _read_fsid unparsable uuid 2024-09-17T16:06:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/6/keyring 2024-09-17T16:06:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:06:04.303 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2024-09-17T16:06:04.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-17T16:06:04.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:06:04.839 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2024-09-17T16:06:04.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-17T16:06:04.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:06:04.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:06:04.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:06:04.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:06:04.866 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:06:04.864+0000 7fd7fde16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:06:04.877 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:06:04.875+0000 7fd7fde16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:06:04.878 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:06:04.876+0000 7fd7fde16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:06:05.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-17T16:06:05.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:06:05.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-17T16:06:05.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:06:05.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:06:05.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:05.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:06:05.276 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:06:05.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:05.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:05.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:06.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:06.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:06.703 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:06:06.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:06:06.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:06.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:06.748 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:06:06.746+0000 7fd7fde16640 -1 Falling back to public interface 2024-09-17T16:06:07.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:08.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:08.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:08.137 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:06:08.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:06:08.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:08.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:08.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:09.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:09.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:09.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:06:09.572 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:06:09.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:09.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:09.866 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:06:09.864+0000 7fd7fde16640 -1 osd.6 0 log_to_monitors true 2024-09-17T16:06:09.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:10.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:10.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:10.967 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:06:10.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:06:10.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:10.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:11.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:12.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:12.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:12.534 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:06:12.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:06:12.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:12.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:12.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:13.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:13.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:13.934 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:06:13.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:06:13.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:13.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:14.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:06:15.333 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:06:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:15.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:15.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:16.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:16.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:16.793 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:06:16.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:06:16.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:16.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:17.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:18.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:18.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:18.267 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:06:18.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:06:18.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:18.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:18.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:19.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:19.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:19.766 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:06:19.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:06:19.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:19.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:20.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:21.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:21.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:21.176 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:06:21.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:06:21.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:21.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:21.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:22.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:22.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:22.618 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:06:22.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:06:22.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:22.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:23.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:24.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:24.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:24.372 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:06:24.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:06:24.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:24.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:25.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:26.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:26.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:26.250 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:06:26.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:06:26.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:26.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:27.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:28.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:28.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:28.454 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:06:28.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:06:28.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:28.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:29.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:30.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:30.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:30.574 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:06:30.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:06:30.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:30.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:31.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:32.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:32.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:32.345 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:06:32.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:06:32.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:32.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:32.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:33.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:33.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:33.914 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:06:33.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:06:33.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:33.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:06:34.394 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1764555180,v1:127.0.0.1:6851/1764555180] [v2:127.0.0.1:6852/1764555180,v1:127.0.0.1:6853/1764555180] exists,up 847dfb63-8a98-445f-976b-8f106467502e 2024-09-17T16:06:34.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:06:34.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:06:34.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:06:34.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-17T16:06:34.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 7 2024-09-17T16:06:34.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-17T16:06:34.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:06:34.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-17T16:06:34.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:06:34.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/7 2024-09-17T16:06:34.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:06:34.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:06:34.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:06:34.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:06:34.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/7' 2024-09-17T16:06:34.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/7/journal' 2024-09-17T16:06:34.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:06:34.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:06:34.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-17T16:06:34.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:06:34.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:06:34.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:06:34.400 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:06:34.401 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:06:34.401 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:06:34.401 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:06:34.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:06:34.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:06:34.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:06:34.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:06:34.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-17T16:06:34.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-17T16:06:34.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:06:34.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:06:34.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:06:34.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:06:34.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:06:34.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:06:34.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/7 2024-09-17T16:06:34.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:06:34.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0a00b894-eea2-4953-92aa-d97b836dc6c1 2024-09-17T16:06:34.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 0a00b894-eea2-4953-92aa-d97b836dc6c1' 2024-09-17T16:06:34.402 INFO:tasks.workunit.client.0.smithi162.stdout:add osd7 0a00b894-eea2-4953-92aa-d97b836dc6c1 2024-09-17T16:06:34.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:06:34.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAKqelmmXGtGBAAyPdt6ivMPPbh9lLX/6S5Vw== 2024-09-17T16:06:34.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAKqelmmXGtGBAAyPdt6ivMPPbh9lLX/6S5Vw=="}' 2024-09-17T16:06:34.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0a00b894-eea2-4953-92aa-d97b836dc6c1 -i td/test-erasure-code/7/new.json 2024-09-17T16:06:34.977 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:06:34.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/7/new.json 2024-09-17T16:06:34.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAKqelmmXGtGBAAyPdt6ivMPPbh9lLX/6S5Vw== --osd-uuid 0a00b894-eea2-4953-92aa-d97b836dc6c1 2024-09-17T16:06:35.021 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:06:35.019+0000 7fb68cdbf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:06:35.065 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:06:35.063+0000 7fb68cdbf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:06:35.066 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:06:35.064+0000 7fb68cdbf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:06:35.066 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:06:35.064+0000 7fb68cdbf640 -1 bdev(0x55763d7fbc00 td/test-erasure-code/7/block) open stat got: (1) Operation not permitted 2024-09-17T16:06:35.066 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:06:35.064+0000 7fb68cdbf640 -1 bluestore(td/test-erasure-code/7) _read_fsid unparsable uuid 2024-09-17T16:06:41.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/7/keyring 2024-09-17T16:06:41.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:06:41.021 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd7 key to auth repository 2024-09-17T16:06:41.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-17T16:06:41.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:06:41.559 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.7 2024-09-17T16:06:41.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-17T16:06:41.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:06:41.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:06:41.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:06:41.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:06:41.585 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:06:41.583+0000 7fd49ee44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:06:41.598 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:06:41.596+0000 7fd49ee44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:06:41.599 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:06:41.597+0000 7fd49ee44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:06:41.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-17T16:06:41.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:06:41.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-17T16:06:41.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:06:41.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:06:41.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:41.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:06:41.972 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:06:41.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:41.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:06:42.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:06:43.198+0000 7fd49ee44640 -1 Falling back to public interface 2024-09-17T16:06:43.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:43.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:43.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:06:43.398 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:06:43.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:43.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:06:43.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:44.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:44.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:44.833 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:06:44.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:06:44.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:44.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:06:45.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:46.105 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:06:46.104+0000 7fd49ee44640 -1 osd.7 0 log_to_monitors true 2024-09-17T16:06:46.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:46.269 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:06:46.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:46.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:06:46.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:46.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:06:46.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:47.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:47.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:47.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:06:47.803 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:06:47.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:47.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:06:48.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:49.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:49.224 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:06:49.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:49.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:06:49.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:49.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:06:49.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:50.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:50.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:50.641 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:06:50.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:06:50.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:50.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:06:51.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:52.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:52.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:52.096 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:06:52.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:06:52.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:52.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:06:52.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:53.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:53.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:53.532 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:06:53.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:06:53.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:53.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:06:54.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:55.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:55.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:55.048 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:06:55.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:06:55.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:55.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:06:55.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:56.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:56.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:56.513 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:06:56.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:06:56.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:56.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:06:57.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:58.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:58.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:58.019 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:06:58.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:06:58.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:58.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:06:58.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:06:59.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:06:59.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:06:59.450 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:06:59.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:06:59.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:06:59.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:07:00.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:01.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:01.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:01.323 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:07:01.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:07:01.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:01.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:07:01.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:02.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:02.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:02.978 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:07:02.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:07:02.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:02.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:07:04.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:05.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:05.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:05.174 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:07:05.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:07:05.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:05.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:07:06.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:07.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:07.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:07.119 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:07:07.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:07:07.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:07.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:07:07.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:08.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:08.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:08.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:07:08.776 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:07:08.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:08.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:07:09.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:10.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:10.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:10.430 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:07:10.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:07:10.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:10.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:07:10.857 INFO:tasks.workunit.client.0.smithi162.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2133876253,v1:127.0.0.1:6859/2133876253] [v2:127.0.0.1:6860/2133876253,v1:127.0.0.1:6861/2133876253] exists,up 0a00b894-eea2-4953-92aa-d97b836dc6c1 2024-09-17T16:07:10.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:07:10.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:07:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:07:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-17T16:07:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 8 2024-09-17T16:07:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-17T16:07:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:07:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2024-09-17T16:07:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:07:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/8 2024-09-17T16:07:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:07:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:07:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:07:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:07:10.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/8' 2024-09-17T16:07:10.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/8/journal' 2024-09-17T16:07:10.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:07:10.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:07:10.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-17T16:07:10.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:07:10.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:07:10.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:07:10.863 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:07:10.863 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:07:10.863 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:07:10.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:07:10.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:07:10.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:07:10.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:07:10.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:07:10.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-17T16:07:10.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-17T16:07:10.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:07:10.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:07:10.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:07:10.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:07:10.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:07:10.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:07:10.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/8 2024-09-17T16:07:10.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:07:10.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=822ca1e5-2ec7-4b53-a0d7-974c23dab20c 2024-09-17T16:07:10.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 822ca1e5-2ec7-4b53-a0d7-974c23dab20c' 2024-09-17T16:07:10.865 INFO:tasks.workunit.client.0.smithi162.stdout:add osd8 822ca1e5-2ec7-4b53-a0d7-974c23dab20c 2024-09-17T16:07:10.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:07:10.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAuqelmEoBINBAAUruzPG0X4b+ECdAqazZDXA== 2024-09-17T16:07:10.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAuqelmEoBINBAAUruzPG0X4b+ECdAqazZDXA=="}' 2024-09-17T16:07:10.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 822ca1e5-2ec7-4b53-a0d7-974c23dab20c -i td/test-erasure-code/8/new.json 2024-09-17T16:07:11.372 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:07:11.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/8/new.json 2024-09-17T16:07:11.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAuqelmEoBINBAAUruzPG0X4b+ECdAqazZDXA== --osd-uuid 822ca1e5-2ec7-4b53-a0d7-974c23dab20c 2024-09-17T16:07:11.413 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:11.410+0000 7fa417615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:07:11.433 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:11.431+0000 7fa417615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:07:11.434 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:11.432+0000 7fa417615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:07:11.435 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:11.432+0000 7fa417615640 -1 bdev(0x5584a9b95c00 td/test-erasure-code/8/block) open stat got: (1) Operation not permitted 2024-09-17T16:07:11.435 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:11.432+0000 7fa417615640 -1 bluestore(td/test-erasure-code/8) _read_fsid unparsable uuid 2024-09-17T16:07:17.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/8/keyring 2024-09-17T16:07:17.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:07:17.501 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd8 key to auth repository 2024-09-17T16:07:17.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2024-09-17T16:07:17.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:07:18.045 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.8 2024-09-17T16:07:18.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2024-09-17T16:07:18.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:07:18.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:07:18.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:07:18.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:07:18.071 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:18.070+0000 7fa12be32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:07:18.101 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:18.099+0000 7fa12be32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:07:18.102 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:18.101+0000 7fa12be32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:07:18.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2024-09-17T16:07:18.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:07:18.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2024-09-17T16:07:18.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:07:18.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:07:18.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:18.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:07:18.467 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:07:18.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:18.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:18.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:19.452 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:19.451+0000 7fa12be32640 -1 Falling back to public interface 2024-09-17T16:07:19.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:19.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:19.926 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:07:19.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:07:19.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:19.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:20.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:21.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:21.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:21.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:07:21.362 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:07:21.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:21.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:21.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:22.433 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:22.432+0000 7fa12be32640 -1 osd.8 0 log_to_monitors true 2024-09-17T16:07:22.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:22.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:22.767 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:07:22.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:07:22.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:22.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:23.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:24.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:24.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:24.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:07:24.397 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:07:24.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:24.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:24.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:25.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:25.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:25.799 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:07:25.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:07:25.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:25.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:26.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:27.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:27.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:27.231 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:07:27.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:07:27.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:27.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:27.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:28.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:28.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:28.755 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:07:28.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:07:28.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:28.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:29.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:30.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:30.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:30.231 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:07:30.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:07:30.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:30.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:30.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:31.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:31.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:31.652 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:07:31.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:07:31.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:31.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:32.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:33.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:33.104 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:07:33.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:33.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:07:33.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:33.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:33.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:34.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:34.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:34.628 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:07:34.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:07:34.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:34.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:35.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:36.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:36.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:36.045 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:07:36.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:07:36.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:36.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:36.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:37.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:37.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:37.972 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:07:37.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:07:37.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:37.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:39.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:40.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:40.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:40.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:07:40.055 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:07:40.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:40.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:40.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:41.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:41.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:41.495 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:07:41.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:07:41.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:41.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:42.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:43.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:43.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:43.080 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:07:43.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:07:43.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:43.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:43.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:44.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:44.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:44.563 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:07:44.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:07:44.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:44.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:45.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:46.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:46.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:46.470 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:07:46.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:07:46.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:46.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:07:46.950 INFO:tasks.workunit.client.0.smithi162.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/1911162967,v1:127.0.0.1:6867/1911162967] [v2:127.0.0.1:6868/1911162967,v1:127.0.0.1:6869/1911162967] exists,up 822ca1e5-2ec7-4b53-a0d7-974c23dab20c 2024-09-17T16:07:46.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:07:46.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:07:46.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:07:46.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-17T16:07:46.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 9 2024-09-17T16:07:46.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-17T16:07:46.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:07:46.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2024-09-17T16:07:46.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:07:46.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/9 2024-09-17T16:07:46.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:07:46.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:07:46.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:07:46.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:07:46.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-09-17T16:07:46.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-09-17T16:07:46.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:07:46.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:07:46.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-17T16:07:46.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:07:46.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:07:46.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:07:46.956 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:07:46.956 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:07:46.956 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:07:46.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:07:46.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:07:46.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:07:46.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:07:46.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:07:46.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-17T16:07:46.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-17T16:07:46.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:07:46.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:07:46.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:07:46.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:07:46.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:07:46.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:07:46.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/9 2024-09-17T16:07:46.958 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:07:46.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b2db456f-7e44-4dbd-85f8-de0b48f51a42 2024-09-17T16:07:46.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 b2db456f-7e44-4dbd-85f8-de0b48f51a42' 2024-09-17T16:07:46.958 INFO:tasks.workunit.client.0.smithi162.stdout:add osd9 b2db456f-7e44-4dbd-85f8-de0b48f51a42 2024-09-17T16:07:46.958 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:07:46.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBSqelmEibeORAAfL+HS/nUSvJR5VMKkmr2/g== 2024-09-17T16:07:46.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBSqelmEibeORAAfL+HS/nUSvJR5VMKkmr2/g=="}' 2024-09-17T16:07:46.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b2db456f-7e44-4dbd-85f8-de0b48f51a42 -i td/test-erasure-code/9/new.json 2024-09-17T16:07:47.491 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:07:47.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/9/new.json 2024-09-17T16:07:47.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBSqelmEibeORAAfL+HS/nUSvJR5VMKkmr2/g== --osd-uuid b2db456f-7e44-4dbd-85f8-de0b48f51a42 2024-09-17T16:07:47.532 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:47.530+0000 7f7c90f2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:07:47.542 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:47.541+0000 7f7c90f2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:07:47.543 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:47.542+0000 7f7c90f2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:07:47.544 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:47.542+0000 7f7c90f2c640 -1 bdev(0x55e602e71c00 td/test-erasure-code/9/block) open stat got: (1) Operation not permitted 2024-09-17T16:07:47.544 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:47.542+0000 7f7c90f2c640 -1 bluestore(td/test-erasure-code/9) _read_fsid unparsable uuid 2024-09-17T16:07:53.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/9/keyring 2024-09-17T16:07:53.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:07:53.181 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd9 key to auth repository 2024-09-17T16:07:53.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2024-09-17T16:07:53.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:07:53.726 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.9 2024-09-17T16:07:53.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2024-09-17T16:07:53.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:07:53.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:07:53.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:07:53.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:07:53.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:53.750+0000 7fd134d4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:07:53.765 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:53.762+0000 7fd134d4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:07:53.766 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:53.763+0000 7fd134d4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:07:54.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2024-09-17T16:07:54.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:07:54.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-17T16:07:54.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:07:54.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:07:54.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:54.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:07:54.145 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:07:54.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:54.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:07:54.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:55.372 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:55.369+0000 7fd134d4d640 -1 Falling back to public interface 2024-09-17T16:07:55.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:55.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:55.575 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:07:55.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:07:55.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:55.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:07:56.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:57.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:07:57.002 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:07:57.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:57.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:07:57.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:07:58.298 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:07:58.295+0000 7fd134d4d640 -1 osd.9 0 log_to_monitors true 2024-09-17T16:07:58.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:07:58.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:07:58.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:07:58.438 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:07:58.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:07:58.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:07:59.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:00.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:00.062 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:08:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:08:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:00.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:01.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:01.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:01.543 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:08:01.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:08:01.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:01.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:01.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:02.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:02.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:02.962 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:08:02.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:08:02.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:02.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:03.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:04.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:04.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:04.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:08:04.396 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:08:04.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:04.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:04.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:05.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:05.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:05.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:08:05.799 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:08:05.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:05.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:06.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:07.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:07.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:07.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:08:07.237 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:08:07.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:07.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:07.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:08.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:08.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:08.644 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:08:08.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:08:08.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:08.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:09.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:10.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:10.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:10.070 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:08:10.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:08:10.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:10.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:10.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:11.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:11.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:11.469 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:08:11.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:08:11.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:11.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:12.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:13.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:13.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:13.630 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:08:13.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:08:13.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:13.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:14.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:15.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:15.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:15.153 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:08:15.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:08:15.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:15.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:16.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:17.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:17.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:17.014 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:08:17.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:08:17.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:17.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:17.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:18.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:18.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:18.968 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:08:18.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:08:18.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:18.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:19.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:20.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:20.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:20.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:08:20.914 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:08:20.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:20.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:21.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:22.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:22.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:22.909 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:08:22.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:08:22.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:22.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:23.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:24.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:24.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:24.424 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-09-17T16:08:24.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T16:08:24.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:24.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:08:24.914 INFO:tasks.workunit.client.0.smithi162.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2952402623,v1:127.0.0.1:6875/2952402623] [v2:127.0.0.1:6876/2952402623,v1:127.0.0.1:6877/2952402623] exists,up b2db456f-7e44-4dbd-85f8-de0b48f51a42 2024-09-17T16:08:24.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:08:24.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:08:24.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:08:24.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-17T16:08:24.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 10 2024-09-17T16:08:24.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-17T16:08:24.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:08:24.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=10 2024-09-17T16:08:24.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:08:24.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/10 2024-09-17T16:08:24.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:08:24.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:08:24.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:08:24.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:08:24.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/10' 2024-09-17T16:08:24.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/10/journal' 2024-09-17T16:08:24.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:08:24.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:08:24.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-17T16:08:24.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:08:24.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:08:24.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:08:24.920 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:08:24.920 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:08:24.920 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:08:24.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:08:24.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:08:24.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:08:24.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:08:24.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:08:24.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-17T16:08:24.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-17T16:08:24.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:08:24.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:08:24.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:08:24.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:08:24.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:08:24.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:08:24.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/10 2024-09-17T16:08:24.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:08:24.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=39d71e25-a4be-4b74-96a9-9cd94dfea0c2 2024-09-17T16:08:24.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd10 39d71e25-a4be-4b74-96a9-9cd94dfea0c2' 2024-09-17T16:08:24.922 INFO:tasks.workunit.client.0.smithi162.stdout:add osd10 39d71e25-a4be-4b74-96a9-9cd94dfea0c2 2024-09-17T16:08:24.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:08:24.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB4qelmXmC5NxAAWOvm6ZkKMrzNF+wmHRY2jw== 2024-09-17T16:08:24.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB4qelmXmC5NxAAWOvm6ZkKMrzNF+wmHRY2jw=="}' 2024-09-17T16:08:24.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 39d71e25-a4be-4b74-96a9-9cd94dfea0c2 -i td/test-erasure-code/10/new.json 2024-09-17T16:08:25.551 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:08:25.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/10/new.json 2024-09-17T16:08:25.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 10 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB4qelmXmC5NxAAWOvm6ZkKMrzNF+wmHRY2jw== --osd-uuid 39d71e25-a4be-4b74-96a9-9cd94dfea0c2 2024-09-17T16:08:25.591 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:08:25.590+0000 7f2c339db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:08:25.593 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:08:25.592+0000 7f2c339db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:08:25.594 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:08:25.593+0000 7f2c339db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:08:25.595 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:08:25.593+0000 7f2c339db640 -1 bdev(0x55a1db7c3c00 td/test-erasure-code/10/block) open stat got: (1) Operation not permitted 2024-09-17T16:08:25.595 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:08:25.593+0000 7f2c339db640 -1 bluestore(td/test-erasure-code/10) _read_fsid unparsable uuid 2024-09-17T16:08:31.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/10/keyring 2024-09-17T16:08:31.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:08:31.701 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd10 key to auth repository 2024-09-17T16:08:31.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd10 key to auth repository 2024-09-17T16:08:31.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:08:32.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.10 2024-09-17T16:08:32.243 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.10 2024-09-17T16:08:32.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 10 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:08:32.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:08:32.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:08:32.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:08:32.270 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:08:32.268+0000 7f7f9a00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:08:32.281 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:08:32.279+0000 7f7f9a00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:08:32.282 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:08:32.281+0000 7f7f9a00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:08:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 10 2024-09-17T16:08:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:08:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=10 2024-09-17T16:08:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:08:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:08:32.697 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:08:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:08:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:33.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:33.878 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:08:33.876+0000 7f7f9a00d640 -1 Falling back to public interface 2024-09-17T16:08:34.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:34.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:34.087 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:08:34.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:08:34.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:34.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:34.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:35.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:35.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:35.526 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:08:35.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:08:35.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:35.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:35.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:36.782 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:08:36.780+0000 7f7f9a00d640 -1 osd.10 0 log_to_monitors true 2024-09-17T16:08:36.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:36.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:36.921 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:08:36.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:08:36.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:36.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:37.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:38.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:38.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:38.369 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:08:38.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:08:38.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:38.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:38.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:39.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:39.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:39.825 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:08:39.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:08:39.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:39.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:40.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:41.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:41.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:41.244 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:08:41.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:08:41.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:41.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:41.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:42.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:42.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:42.680 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:08:42.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:08:42.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:42.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:43.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:44.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:44.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:44.076 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:08:44.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:08:44.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:44.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:44.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:45.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:45.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:45.494 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:08:45.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:08:45.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:45.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:45.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:46.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:46.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:46.997 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:08:46.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:08:46.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:46.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:47.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:48.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:48.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:48.478 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:08:48.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:08:48.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:48.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:49.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:50.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:50.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:50.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:08:50.036 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:08:50.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:50.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:50.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:51.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:51.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:51.500 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:08:51.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:08:51.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:51.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:52.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:53.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:53.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:53.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:08:53.238 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:08:53.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:53.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:54.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:55.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:55.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:55.025 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:08:55.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:08:55.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:55.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:55.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:56.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:56.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:56.883 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:08:56.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:08:56.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:56.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:58.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:08:59.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:08:59.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:08:59.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:08:59.013 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:08:59.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:08:59.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:08:59.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:09:00.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:09:00.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:09:00.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:09:00.449 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:09:00.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:09:00.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T16:09:00.947 INFO:tasks.workunit.client.0.smithi162.stdout:osd.10 up in weight 1 up_from 55 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/3090069836,v1:127.0.0.1:6883/3090069836] [v2:127.0.0.1:6884/3090069836,v1:127.0.0.1:6885/3090069836] exists,up 39d71e25-a4be-4b74-96a9-9cd94dfea0c2 2024-09-17T16:09:00.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:09:00.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:09:00.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:09:00.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:39: run: create_rbd_pool 2024-09-17T16:09:00.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-17T16:09:01.401 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-09-17T16:09:01.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-17T16:09:01.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T16:09:01.961 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T16:09:01.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:09:02.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-17T16:09:06.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:40: run: wait_for_clean 2024-09-17T16:09:06.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:09:06.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:09:06.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:09:06.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:09:06.768 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:09:06.768 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:09:06.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:09:06.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:09:06.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:09:06.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:09:06.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:09:06.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:09:06.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:09:06.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:09:06.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:09:07.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:09:07.416 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:09:07.416 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:09:07.416 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:09:07.416 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:09:07.417 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:09:07.417 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:09:07.417 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:09:07.417 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-17T16:09:07.417 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:09:07.417 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-17T16:09:07.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:09:07.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:07.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:09:07.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836561 2024-09-17T16:09:07.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836561 2024-09-17T16:09:07.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561' 2024-09-17T16:09:07.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:07.530 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:09:07.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673035 2024-09-17T16:09:07.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673035 2024-09-17T16:09:07.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673035' 2024-09-17T16:09:07.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:07.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:09:07.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509505 2024-09-17T16:09:07.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509505 2024-09-17T16:09:07.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509505' 2024-09-17T16:09:07.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:07.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:09:07.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345983 2024-09-17T16:09:07.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345983 2024-09-17T16:09:07.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509505 3-85899345983' 2024-09-17T16:09:07.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:07.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:09:07.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182452 2024-09-17T16:09:07.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182452 2024-09-17T16:09:07.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509505 3-85899345983 4-107374182452' 2024-09-17T16:09:07.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:07.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:09:08.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018924 2024-09-17T16:09:08.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018924 2024-09-17T16:09:08.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509505 3-85899345983 4-107374182452 5-128849018924' 2024-09-17T16:09:08.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:08.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:09:08.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855396 2024-09-17T16:09:08.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855396 2024-09-17T16:09:08.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509505 3-85899345983 4-107374182452 5-128849018924 6-150323855396' 2024-09-17T16:09:08.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:08.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T16:09:08.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691869 2024-09-17T16:09:08.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691869 2024-09-17T16:09:08.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509505 3-85899345983 4-107374182452 5-128849018924 6-150323855396 7-171798691869' 2024-09-17T16:09:08.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:08.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T16:09:08.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528344 2024-09-17T16:09:08.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528344 2024-09-17T16:09:08.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509505 3-85899345983 4-107374182452 5-128849018924 6-150323855396 7-171798691869 8-193273528344' 2024-09-17T16:09:08.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:08.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T16:09:08.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364813 2024-09-17T16:09:08.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364813 2024-09-17T16:09:08.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509505 3-85899345983 4-107374182452 5-128849018924 6-150323855396 7-171798691869 8-193273528344 9-214748364813' 2024-09-17T16:09:08.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:08.543 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-17T16:09:08.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201293 2024-09-17T16:09:08.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201293 2024-09-17T16:09:08.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509505 3-85899345983 4-107374182452 5-128849018924 6-150323855396 7-171798691869 8-193273528344 9-214748364813 10-236223201293' 2024-09-17T16:09:08.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:08.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836561 2024-09-17T16:09:08.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:08.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:09:08.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836561 2024-09-17T16:09:08.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:08.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836561 2024-09-17T16:09:08.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2024-09-17T16:09:08.656 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836561 2024-09-17T16:09:08.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:09:09.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836561 -lt 21474836561 2024-09-17T16:09:09.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:09.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673035 2024-09-17T16:09:09.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:09.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:09:09.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673035 2024-09-17T16:09:09.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:09.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673035 2024-09-17T16:09:09.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-09-17T16:09:09.055 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949673035 2024-09-17T16:09:09.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:09:09.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673036 -lt 42949673035 2024-09-17T16:09:09.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:09.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509505 2024-09-17T16:09:09.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:09.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:09:09.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509505 2024-09-17T16:09:09.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:09.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509505 2024-09-17T16:09:09.498 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509505 2024-09-17T16:09:09.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2024-09-17T16:09:09.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:09:09.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509505 -lt 64424509505 2024-09-17T16:09:09.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:09.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345983 2024-09-17T16:09:09.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:09.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:09:09.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345983 2024-09-17T16:09:09.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:09.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345983 2024-09-17T16:09:09.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345983' 2024-09-17T16:09:09.934 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345983 2024-09-17T16:09:09.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:09:10.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345983 -lt 85899345983 2024-09-17T16:09:10.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:10.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182452 2024-09-17T16:09:10.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:10.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:09:10.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182452 2024-09-17T16:09:10.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:10.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182452 2024-09-17T16:09:10.322 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182452 2024-09-17T16:09:10.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182452' 2024-09-17T16:09:10.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:09:10.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182452 -lt 107374182452 2024-09-17T16:09:10.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:10.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018924 2024-09-17T16:09:10.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:10.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:09:10.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018924 2024-09-17T16:09:10.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:10.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018924 2024-09-17T16:09:10.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018924' 2024-09-17T16:09:10.753 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 128849018924 2024-09-17T16:09:10.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:09:11.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018924 -lt 128849018924 2024-09-17T16:09:11.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:11.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855396 2024-09-17T16:09:11.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:11.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:09:11.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855396 2024-09-17T16:09:11.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:11.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855396 2024-09-17T16:09:11.206 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 150323855396 2024-09-17T16:09:11.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855396' 2024-09-17T16:09:11.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:09:11.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855397 -lt 150323855396 2024-09-17T16:09:11.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:11.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691869 2024-09-17T16:09:11.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:11.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T16:09:11.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691869 2024-09-17T16:09:11.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:11.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691869 2024-09-17T16:09:11.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691869' 2024-09-17T16:09:11.592 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691869 2024-09-17T16:09:11.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T16:09:11.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691869 -lt 171798691869 2024-09-17T16:09:11.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:11.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528344 2024-09-17T16:09:11.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:11.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T16:09:11.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528344 2024-09-17T16:09:11.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:11.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528344 2024-09-17T16:09:11.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528344' 2024-09-17T16:09:11.981 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 193273528344 2024-09-17T16:09:11.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T16:09:12.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528344 -lt 193273528344 2024-09-17T16:09:12.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:12.425 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364813 2024-09-17T16:09:12.425 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:12.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T16:09:12.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364813 2024-09-17T16:09:12.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:12.428 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.9 seq 214748364813 2024-09-17T16:09:12.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364813 2024-09-17T16:09:12.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364813' 2024-09-17T16:09:12.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T16:09:12.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364814 -lt 214748364813 2024-09-17T16:09:12.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:12.857 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201293 2024-09-17T16:09:12.857 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:12.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-17T16:09:12.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201293 2024-09-17T16:09:12.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201293 2024-09-17T16:09:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201293' 2024-09-17T16:09:12.860 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.10 seq 236223201293 2024-09-17T16:09:12.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-17T16:09:13.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201294 -lt 236223201293 2024-09-17T16:09:13.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:09:13.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:09:13.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:09:13.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T16:09:13.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:09:13.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:09:13.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:09:13.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:09:13.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:09:13.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:09:13.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:09:14.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T16:09:14.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:09:14.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:09:14.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:09:14.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T16:09:14.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:09:14.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:09:14.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: get_asok_path osd.0 2024-09-17T16:09:14.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:09:14.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:09:14.923 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:09:14.923 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:09:14.923 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:09:14.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56062/ceph-osd.0.asok 2024-09-17T16:09:14.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: CEPH_ARGS= 2024-09-17T16:09:14.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: ceph --admin-daemon /tmp/ceph-asok.56062/ceph-osd.0.asok log flush 2024-09-17T16:09:15.001 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:09:15.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:43: run: grep 'load: jerasure.*lrc' td/test-erasure-code/osd.0.log 2024-09-17T16:09:15.012 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:02:32.602+0000 7f77937a8640 0 load: jerasure load: lrc 2024-09-17T16:09:15.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:44: run: create_erasure_coded_pool ecpool 2024-09-17T16:09:15.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:56: create_erasure_coded_pool: local poolname=ecpool 2024-09-17T16:09:15.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:58: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd 2024-09-17T16:09:15.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:60: create_erasure_coded_pool: create_pool ecpool 12 12 erasure myprofile 2024-09-17T16:09:15.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 12 12 erasure myprofile 2024-09-17T16:09:16.477 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' already exists 2024-09-17T16:09:16.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:09:17.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:62: create_erasure_coded_pool: wait_for_clean 2024-09-17T16:09:18.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:09:18.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:09:18.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:09:18.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:09:18.496 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:09:18.496 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:09:18.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:09:18.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:09:18.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:09:18.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:09:18.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:09:18.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:09:18.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:09:18.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:09:18.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:09:18.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:09:18.569 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:09:18.569 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:09:18.569 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:09:18.569 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:09:18.569 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:09:18.569 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:09:18.569 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:09:18.569 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-17T16:09:18.569 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:09:18.569 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-17T16:09:18.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:09:18.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:18.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:09:18.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836565 2024-09-17T16:09:18.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836565 2024-09-17T16:09:18.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565' 2024-09-17T16:09:18.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:18.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:09:19.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673040 2024-09-17T16:09:19.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673040 2024-09-17T16:09:19.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673040' 2024-09-17T16:09:19.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:19.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:09:19.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509509 2024-09-17T16:09:19.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509509 2024-09-17T16:09:19.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509509' 2024-09-17T16:09:19.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:19.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:09:19.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345987 2024-09-17T16:09:19.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345987 2024-09-17T16:09:19.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509509 3-85899345987' 2024-09-17T16:09:19.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:19.668 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:09:19.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182456 2024-09-17T16:09:19.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182456 2024-09-17T16:09:19.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509509 3-85899345987 4-107374182456' 2024-09-17T16:09:19.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:19.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:09:20.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018928 2024-09-17T16:09:20.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018928 2024-09-17T16:09:20.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509509 3-85899345987 4-107374182456 5-128849018928' 2024-09-17T16:09:20.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:20.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:09:20.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855401 2024-09-17T16:09:20.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855401 2024-09-17T16:09:20.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509509 3-85899345987 4-107374182456 5-128849018928 6-150323855401' 2024-09-17T16:09:20.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:20.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T16:09:20.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691873 2024-09-17T16:09:20.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691873 2024-09-17T16:09:20.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509509 3-85899345987 4-107374182456 5-128849018928 6-150323855401 7-171798691873' 2024-09-17T16:09:20.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:20.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T16:09:20.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528348 2024-09-17T16:09:20.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528348 2024-09-17T16:09:20.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509509 3-85899345987 4-107374182456 5-128849018928 6-150323855401 7-171798691873 8-193273528348' 2024-09-17T16:09:20.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:20.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T16:09:20.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364818 2024-09-17T16:09:20.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364818 2024-09-17T16:09:20.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509509 3-85899345987 4-107374182456 5-128849018928 6-150323855401 7-171798691873 8-193273528348 9-214748364818' 2024-09-17T16:09:20.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:20.577 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-17T16:09:20.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201297 2024-09-17T16:09:20.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201297 2024-09-17T16:09:20.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509509 3-85899345987 4-107374182456 5-128849018928 6-150323855401 7-171798691873 8-193273528348 9-214748364818 10-236223201297' 2024-09-17T16:09:20.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:20.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836565 2024-09-17T16:09:20.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:20.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:09:20.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836565 2024-09-17T16:09:20.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:20.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836565 2024-09-17T16:09:20.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-09-17T16:09:20.694 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836565 2024-09-17T16:09:20.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:09:21.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836566 -lt 21474836565 2024-09-17T16:09:21.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:21.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673040 2024-09-17T16:09:21.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:21.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:09:21.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673040 2024-09-17T16:09:21.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:21.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673040 2024-09-17T16:09:21.086 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949673040 2024-09-17T16:09:21.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-09-17T16:09:21.087 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:09:21.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673040 -lt 42949673040 2024-09-17T16:09:21.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:21.669 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509509 2024-09-17T16:09:21.669 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:21.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:09:21.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509509 2024-09-17T16:09:21.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:21.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509509 2024-09-17T16:09:21.672 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509509 2024-09-17T16:09:21.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509509' 2024-09-17T16:09:21.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:09:22.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509509 -lt 64424509509 2024-09-17T16:09:22.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:22.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345987 2024-09-17T16:09:22.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:22.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:09:22.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345987 2024-09-17T16:09:22.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:22.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345987 2024-09-17T16:09:22.107 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345987 2024-09-17T16:09:22.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345987' 2024-09-17T16:09:22.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:09:22.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345987 -lt 85899345987 2024-09-17T16:09:22.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:22.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182456 2024-09-17T16:09:22.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:09:22.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182456 2024-09-17T16:09:22.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:22.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182456 2024-09-17T16:09:22.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182456' 2024-09-17T16:09:22.497 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182456 2024-09-17T16:09:22.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:09:22.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182456 -lt 107374182456 2024-09-17T16:09:22.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:22.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018928 2024-09-17T16:09:22.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:22.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:09:22.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018928 2024-09-17T16:09:22.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:22.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018928 2024-09-17T16:09:22.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018928' 2024-09-17T16:09:22.951 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 128849018928 2024-09-17T16:09:22.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:09:23.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018929 -lt 128849018928 2024-09-17T16:09:23.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:23.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855401 2024-09-17T16:09:23.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:23.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:09:23.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855401 2024-09-17T16:09:23.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:23.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855401 2024-09-17T16:09:23.412 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 150323855401 2024-09-17T16:09:23.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855401' 2024-09-17T16:09:23.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:09:23.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855401 -lt 150323855401 2024-09-17T16:09:23.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:23.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691873 2024-09-17T16:09:23.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:23.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T16:09:23.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691873 2024-09-17T16:09:23.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:23.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691873 2024-09-17T16:09:23.839 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691873 2024-09-17T16:09:23.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691873' 2024-09-17T16:09:23.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T16:09:24.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691874 -lt 171798691873 2024-09-17T16:09:24.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:24.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528348 2024-09-17T16:09:24.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:24.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T16:09:24.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528348 2024-09-17T16:09:24.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:24.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528348 2024-09-17T16:09:24.274 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 193273528348 2024-09-17T16:09:24.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528348' 2024-09-17T16:09:24.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T16:09:24.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528349 -lt 193273528348 2024-09-17T16:09:24.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:24.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364818 2024-09-17T16:09:24.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:24.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T16:09:24.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364818 2024-09-17T16:09:24.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:24.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364818 2024-09-17T16:09:24.709 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.9 seq 214748364818 2024-09-17T16:09:24.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364818' 2024-09-17T16:09:24.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T16:09:25.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364819 -lt 214748364818 2024-09-17T16:09:25.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:25.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201297 2024-09-17T16:09:25.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:25.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-17T16:09:25.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201297 2024-09-17T16:09:25.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:25.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201297 2024-09-17T16:09:25.187 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.10 seq 236223201297 2024-09-17T16:09:25.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201297' 2024-09-17T16:09:25.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-17T16:09:25.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201298 -lt 236223201297 2024-09-17T16:09:25.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:09:25.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:09:25.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:09:26.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-17T16:09:26.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:09:26.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:09:26.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:09:26.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:09:26.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:09:26.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:09:26.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:09:26.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-17T16:09:26.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:09:26.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:09:26.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:09:27.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-17T16:09:27.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:09:27.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:09:27.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: set 2024-09-17T16:09:27.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-17T16:09:27.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: local 'funcs=TEST_alignment_constraints 2024-09-17T16:09:27.098 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_chunk_mapping 2024-09-17T16:09:27.098 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_put_get_isa 2024-09-17T16:09:27.098 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_put_get_jerasure 2024-09-17T16:09:27.098 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_put_get_lrc_advanced 2024-09-17T16:09:27.099 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_put_get_lrc_kml 2024-09-17T16:09:27.099 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_put_get_shec' 2024-09-17T16:09:27.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-17T16:09:27.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_alignment_constraints td/test-erasure-code 2024-09-17T16:09:27.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: TEST_alignment_constraints: local payload=ABC 2024-09-17T16:09:27.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: TEST_alignment_constraints: echo ABC 2024-09-17T16:09:27.099 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-17T16:09:27.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: local stripe_unit=4096 2024-09-17T16:09:27.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: ceph osd erasure-code-profile get myprofile 2024-09-17T16:09:27.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: grep k= 2024-09-17T16:09:27.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: eval local k=2 2024-09-17T16:09:27.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: local k=2 2024-09-17T16:09:27.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: TEST_alignment_constraints: local block_size=8191 2024-09-17T16:09:27.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:263: TEST_alignment_constraints: dd if=/dev/zero of=td/test-erasure-code/ORIGINAL bs=8191 count=2 2024-09-17T16:09:27.551 INFO:tasks.workunit.client.0.smithi162.stderr:2+0 records in 2024-09-17T16:09:27.551 INFO:tasks.workunit.client.0.smithi162.stderr:2+0 records out 2024-09-17T16:09:27.551 INFO:tasks.workunit.client.0.smithi162.stderr:16382 bytes (16 kB, 16 KiB) copied, 8.8827e-05 s, 184 MB/s 2024-09-17T16:09:27.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:264: TEST_alignment_constraints: rados --block-size=8191 --pool ecpool put UNALIGNED td/test-erasure-code/ORIGINAL 2024-09-17T16:09:27.579 INFO:tasks.workunit.client.0.smithi162.stderr:INFO: op_size has been rounded to 8192 2024-09-17T16:09:28.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: TEST_alignment_constraints: rm td/test-erasure-code/ORIGINAL 2024-09-17T16:09:28.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-17T16:09:28.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_chunk_mapping td/test-erasure-code 2024-09-17T16:09:28.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:304: TEST_chunk_mapping: local dir=td/test-erasure-code 2024-09-17T16:09:28.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:311: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code ecpool 0 1 2024-09-17T16:09:28.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-09-17T16:09:28.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=ecpool 2024-09-17T16:09:28.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=0 2024-09-17T16:09:28.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=1 2024-09-17T16:09:28.020 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-17T16:09:28.020 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-17T16:09:28.037 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-17T16:09:28.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTecpool 2024-09-17T16:09:28.037 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-17T16:09:28.037 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-17T16:09:28.054 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-17T16:09:28.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDecpool 2024-09-17T16:09:28.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTecpool SECONDecpool' 2024-09-17T16:09:28.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTecpool SECONDecpool' 2024-09-17T16:09:28.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool ecpool put SOMETHINGecpool td/test-erasure-code/ORIGINAL 2024-09-17T16:09:28.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool ecpool get SOMETHINGecpool td/test-erasure-code/COPY 2024-09-17T16:09:28.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-09-17T16:09:28.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-17T16:09:28.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGecpool 2024-09-17T16:09:28.427 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-09-17T16:09:28.427 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:09:28.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:09:28.822 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:09:28.822 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:09:28.822 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-09-17T16:09:28.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 2024-09-17T16:09:28.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-09-17T16:09:28.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-09-17T16:09:28.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-09-17T16:09:28.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-17T16:09:28.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.3 flush_journal 2024-09-17T16:09:28.917 INFO:tasks.workunit.client.0.smithi162.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-17T16:09:28.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-17T16:09:28.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-17T16:09:28.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.5 flush_journal 2024-09-17T16:09:29.015 INFO:tasks.workunit.client.0.smithi162.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-17T16:09:29.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-17T16:09:29.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-17T16:09:29.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-09-17T16:09:29.111 INFO:tasks.workunit.client.0.smithi162.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-17T16:09:29.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-17T16:09:29.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-17T16:09:29.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.0 flush_journal 2024-09-17T16:09:29.205 INFO:tasks.workunit.client.0.smithi162.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-17T16:09:29.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-17T16:09:29.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-17T16:09:29.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:09:29.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-09-17T16:09:29.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-09-17T16:09:29.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-17T16:09:29.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGecpool 2024-09-17T16:09:29.209 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-09-17T16:09:29.209 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:09:29.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:09:29.598 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:09:29.598 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:09:29.598 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-09-17T16:09:29.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 2024-09-17T16:09:29.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-09-17T16:09:29.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-09-17T16:09:29.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-17T16:09:29.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2024-09-17T16:09:29.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:09:29.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T16:09:29.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTecpool 2024-09-17T16:09:29.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:09:29.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-17T16:09:29.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-17T16:09:29.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:09:29.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T16:09:29.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:09:29.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.3 2024-09-17T16:09:29.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:09:29.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:09:29.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:09:29.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:09:29.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:09:29.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:09:29.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-17T16:09:29.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-17T16:09:29.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:09:29.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T16:09:29.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:09:29.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/3 2024-09-17T16:09:29.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/3 SOMETHINGecpool get-bytes 2024-09-17T16:09:30.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 3 2024-09-17T16:09:30.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2024-09-17T16:09:30.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:09:30.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T16:09:30.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:09:30.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/3 2024-09-17T16:09:30.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:09:30.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:09:30.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:09:30.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:09:30.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-09-17T16:09:30.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-09-17T16:09:30.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:09:30.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:09:30.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-17T16:09:30.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:09:30.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:09:30.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:09:30.657 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:09:30.658 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:09:30.658 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:09:30.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:09:30.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:09:30.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:09:30.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-17T16:09:30.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-17T16:09:30.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:09:30.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:09:30.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:09:30.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:09:30.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:09:30.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:09:30.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/3 2024-09-17T16:09:30.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T16:09:30.659 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-09-17T16:09:30.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:09:30.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/3/whoami 2024-09-17T16:09:30.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T16:09:30.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:09:30.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:09:30.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:09:30.684 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:09:30.681+0000 7f50b4e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:09:30.805 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:09:30.803+0000 7f50b4e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:09:30.807 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:09:30.804+0000 7f50b4e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:09:31.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T16:09:31.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:09:31.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:09:31.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:09:31.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:09:31.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:09:31.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:09:31.122 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:09:31.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:09:31.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:09:31.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:09:31.937 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:09:31.932+0000 7f50b4e0b640 -1 Falling back to public interface 2024-09-17T16:09:32.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:09:32.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:09:32.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:09:32.513 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:09:32.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:09:32.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:09:32.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:09:33.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:09:33.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:09:33.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:09:33.987 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:09:33.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:09:33.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:09:34.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:09:35.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:09:35.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:09:35.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:09:35.424 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:09:35.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:09:35.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:09:35.697 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:09:35.694+0000 7f50b4e0b640 -1 osd.3 76 log_to_monitors true 2024-09-17T16:09:35.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:09:36.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:09:36.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:09:36.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:09:36.832 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:09:36.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:09:36.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:09:37.526 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 80 up_thru 80 down_at 77 last_clean_interval [20,76) [v2:127.0.0.1:6826/1799052560,v1:127.0.0.1:6827/1799052560] [v2:127.0.0.1:6828/1799052560,v1:127.0.0.1:6829/1799052560] exists,up a1b58765-3583-41df-91a6-d67cf3dd9c17 2024-09-17T16:09:37.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:09:37.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:09:37.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:09:37.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:09:37.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:09:37.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:09:37.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:09:37.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:09:37.527 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:09:37.527 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:09:37.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:09:37.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:09:37.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:09:37.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:09:37.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:09:37.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:09:37.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:09:37.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:09:37.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:09:38.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:09:38.145 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:09:38.145 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:09:38.145 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:09:38.145 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:09:38.145 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:09:38.145 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:09:38.145 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:09:38.145 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-17T16:09:38.145 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:09:38.146 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-17T16:09:38.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:09:38.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:38.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:09:38.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836571 2024-09-17T16:09:38.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836571 2024-09-17T16:09:38.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571' 2024-09-17T16:09:38.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:38.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:09:38.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673045 2024-09-17T16:09:38.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673045 2024-09-17T16:09:38.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673045' 2024-09-17T16:09:38.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:38.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:09:38.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509515 2024-09-17T16:09:38.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509515 2024-09-17T16:09:38.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509515' 2024-09-17T16:09:38.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:38.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:09:38.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383683 2024-09-17T16:09:38.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383683 2024-09-17T16:09:38.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509515 3-343597383683' 2024-09-17T16:09:38.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:38.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:09:38.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182462 2024-09-17T16:09:38.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182462 2024-09-17T16:09:38.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509515 3-343597383683 4-107374182462' 2024-09-17T16:09:38.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:38.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:09:38.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018934 2024-09-17T16:09:38.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018934 2024-09-17T16:09:38.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509515 3-343597383683 4-107374182462 5-128849018934' 2024-09-17T16:09:38.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:38.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:09:38.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855406 2024-09-17T16:09:38.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855406 2024-09-17T16:09:38.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509515 3-343597383683 4-107374182462 5-128849018934 6-150323855406' 2024-09-17T16:09:38.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:38.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T16:09:39.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691879 2024-09-17T16:09:39.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691879 2024-09-17T16:09:39.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509515 3-343597383683 4-107374182462 5-128849018934 6-150323855406 7-171798691879' 2024-09-17T16:09:39.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:39.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T16:09:39.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528354 2024-09-17T16:09:39.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528354 2024-09-17T16:09:39.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509515 3-343597383683 4-107374182462 5-128849018934 6-150323855406 7-171798691879 8-193273528354' 2024-09-17T16:09:39.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:39.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T16:09:39.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364824 2024-09-17T16:09:39.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364824 2024-09-17T16:09:39.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509515 3-343597383683 4-107374182462 5-128849018934 6-150323855406 7-171798691879 8-193273528354 9-214748364824' 2024-09-17T16:09:39.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:39.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-17T16:09:39.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201303 2024-09-17T16:09:39.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201303 2024-09-17T16:09:39.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509515 3-343597383683 4-107374182462 5-128849018934 6-150323855406 7-171798691879 8-193273528354 9-214748364824 10-236223201303' 2024-09-17T16:09:39.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:39.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836571 2024-09-17T16:09:39.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:39.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:09:39.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836571 2024-09-17T16:09:39.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:39.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836571 2024-09-17T16:09:39.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-09-17T16:09:39.390 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836571 2024-09-17T16:09:39.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:09:39.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836571 -lt 21474836571 2024-09-17T16:09:39.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:39.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673045 2024-09-17T16:09:39.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:39.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:09:39.819 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673045 2024-09-17T16:09:39.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:39.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673045 2024-09-17T16:09:39.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-09-17T16:09:39.821 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673045 2024-09-17T16:09:39.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:09:40.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673046 -lt 42949673045 2024-09-17T16:09:40.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:40.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509515 2024-09-17T16:09:40.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:40.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:09:40.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509515 2024-09-17T16:09:40.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:40.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509515 2024-09-17T16:09:40.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509515' 2024-09-17T16:09:40.211 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509515 2024-09-17T16:09:40.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:09:40.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509515 -lt 64424509515 2024-09-17T16:09:40.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:40.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-343597383683 2024-09-17T16:09:40.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:40.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:09:40.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-343597383683 2024-09-17T16:09:40.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:40.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383683 2024-09-17T16:09:40.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 343597383683' 2024-09-17T16:09:40.633 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 343597383683 2024-09-17T16:09:40.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:09:41.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383683 -lt 343597383683 2024-09-17T16:09:41.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:41.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182462 2024-09-17T16:09:41.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:41.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:09:41.026 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182462 2024-09-17T16:09:41.026 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:41.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182462 2024-09-17T16:09:41.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182462' 2024-09-17T16:09:41.028 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182462 2024-09-17T16:09:41.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:09:41.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182462 -lt 107374182462 2024-09-17T16:09:41.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:41.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018934 2024-09-17T16:09:41.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:41.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:09:41.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018934 2024-09-17T16:09:41.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:41.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018934 2024-09-17T16:09:41.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018934' 2024-09-17T16:09:41.470 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 128849018934 2024-09-17T16:09:41.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:09:41.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018934 -lt 128849018934 2024-09-17T16:09:41.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:41.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855406 2024-09-17T16:09:41.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:41.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:09:41.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855406 2024-09-17T16:09:41.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:41.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855406 2024-09-17T16:09:41.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855406' 2024-09-17T16:09:41.905 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 150323855406 2024-09-17T16:09:41.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:09:42.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855407 -lt 150323855406 2024-09-17T16:09:42.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:42.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691879 2024-09-17T16:09:42.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:42.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T16:09:42.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691879 2024-09-17T16:09:42.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:42.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691879 2024-09-17T16:09:42.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-09-17T16:09:42.334 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691879 2024-09-17T16:09:42.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T16:09:42.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691879 -lt 171798691879 2024-09-17T16:09:42.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:42.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528354 2024-09-17T16:09:42.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:42.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T16:09:42.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528354 2024-09-17T16:09:42.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:42.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528354 2024-09-17T16:09:42.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528354' 2024-09-17T16:09:42.761 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528354 2024-09-17T16:09:42.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T16:09:43.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528355 -lt 193273528354 2024-09-17T16:09:43.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:43.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364824 2024-09-17T16:09:43.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T16:09:43.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364824 2024-09-17T16:09:43.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:43.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364824 2024-09-17T16:09:43.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364824' 2024-09-17T16:09:43.213 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 214748364824 2024-09-17T16:09:43.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T16:09:43.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364824 -lt 214748364824 2024-09-17T16:09:43.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:43.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201303 2024-09-17T16:09:43.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-17T16:09:43.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201303 2024-09-17T16:09:43.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201303 2024-09-17T16:09:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201303' 2024-09-17T16:09:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.10 seq 236223201303 2024-09-17T16:09:43.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-17T16:09:44.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201304 -lt 236223201303 2024-09-17T16:09:44.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:09:44.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:09:44.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:09:44.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-17T16:09:44.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:09:44.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:09:44.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:09:44.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:09:44.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:09:44.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:09:44.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:09:45.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-17T16:09:45.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:09:45.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:09:45.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:09:45.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-17T16:09:45.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:09:45.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:09:45.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-17T16:09:45.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2024-09-17T16:09:45.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:09:45.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-17T16:09:45.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:09:45.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-17T16:09:45.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDecpool 2024-09-17T16:09:45.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-17T16:09:45.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:09:45.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-17T16:09:45.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:09:45.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.5 2024-09-17T16:09:45.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:09:45.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:09:45.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:09:45.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:09:45.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:09:45.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:09:45.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-17T16:09:45.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-17T16:09:45.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:09:45.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-17T16:09:45.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:09:45.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/5 2024-09-17T16:09:45.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/5 SOMETHINGecpool get-bytes 2024-09-17T16:09:46.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 5 2024-09-17T16:09:46.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2024-09-17T16:09:46.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:09:46.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-17T16:09:46.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:09:46.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/5 2024-09-17T16:09:46.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:09:46.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:09:46.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:09:46.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:09:46.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-09-17T16:09:46.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-09-17T16:09:46.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:09:46.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:09:46.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-17T16:09:46.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:09:46.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:09:46.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:09:46.711 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:09:46.711 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:09:46.711 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:09:46.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:09:46.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:09:46.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:09:46.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-17T16:09:46.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-17T16:09:46.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:09:46.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:09:46.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:09:46.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:09:46.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:09:46.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:09:46.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/5 2024-09-17T16:09:46.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-17T16:09:46.712 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.5 2024-09-17T16:09:46.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:09:46.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/5/whoami 2024-09-17T16:09:46.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-17T16:09:46.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:09:46.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:09:46.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:09:46.736 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:09:46.734+0000 7f3ea8c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:09:46.738 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:09:46.736+0000 7f3ea8c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:09:46.739 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:09:46.737+0000 7f3ea8c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:09:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-17T16:09:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:09:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T16:09:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:09:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:09:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:09:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:09:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:09:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:09:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:09:47.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:09:48.363 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:09:48.362+0000 7f3ea8c15640 -1 Falling back to public interface 2024-09-17T16:09:48.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:09:48.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:09:48.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:09:48.921 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:09:48.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:09:48.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:09:49.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:09:50.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:09:50.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:09:50.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:09:50.357 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:09:50.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:09:50.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:09:50.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:09:51.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:09:51.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:09:51.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:09:51.793 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:09:51.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:09:51.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:09:51.842 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:09:51.840+0000 7f3ea8c15640 -1 osd.5 82 log_to_monitors true 2024-09-17T16:09:52.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:09:53.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:09:53.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:09:53.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:09:53.195 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:09:53.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:09:53.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:09:53.930 INFO:tasks.workunit.client.0.smithi162.stderr:osd.5 up in weight 1 up_from 86 up_thru 86 down_at 83 last_clean_interval [30,82) [v2:127.0.0.1:6842/3848972771,v1:127.0.0.1:6843/3848972771] [v2:127.0.0.1:6844/3848972771,v1:127.0.0.1:6845/3848972771] exists,up 316d12e1-735e-43da-ba4d-68452198e455 2024-09-17T16:09:53.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:09:53.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:09:53.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:09:53.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:09:53.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:09:53.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:09:53.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:09:53.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:09:53.931 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:09:53.931 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:09:53.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:09:53.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:09:53.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:09:54.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:09:54.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:09:54.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:09:54.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:09:54.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:09:54.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:09:54.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:09:54.817 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:09:54.817 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:09:54.817 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:09:54.817 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:09:54.817 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:09:54.817 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:09:54.817 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:09:54.817 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-17T16:09:54.818 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:09:54.818 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-17T16:09:54.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:09:54.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:54.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:09:54.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836577 2024-09-17T16:09:54.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836577 2024-09-17T16:09:54.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577' 2024-09-17T16:09:54.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:54.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:09:55.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673051 2024-09-17T16:09:55.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673051 2024-09-17T16:09:55.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051' 2024-09-17T16:09:55.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:55.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:09:55.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509520 2024-09-17T16:09:55.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509520 2024-09-17T16:09:55.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509520' 2024-09-17T16:09:55.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:55.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:09:55.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383688 2024-09-17T16:09:55.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383688 2024-09-17T16:09:55.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509520 3-343597383688' 2024-09-17T16:09:55.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:55.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:09:55.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182467 2024-09-17T16:09:55.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182467 2024-09-17T16:09:55.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509520 3-343597383688 4-107374182467' 2024-09-17T16:09:55.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:55.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:09:55.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=369367187459 2024-09-17T16:09:55.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 369367187459 2024-09-17T16:09:55.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509520 3-343597383688 4-107374182467 5-369367187459' 2024-09-17T16:09:55.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:55.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:09:55.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855412 2024-09-17T16:09:55.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855412 2024-09-17T16:09:55.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509520 3-343597383688 4-107374182467 5-369367187459 6-150323855412' 2024-09-17T16:09:55.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:55.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T16:09:55.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691884 2024-09-17T16:09:55.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691884 2024-09-17T16:09:55.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509520 3-343597383688 4-107374182467 5-369367187459 6-150323855412 7-171798691884' 2024-09-17T16:09:55.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:55.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T16:09:55.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528359 2024-09-17T16:09:55.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528359 2024-09-17T16:09:55.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509520 3-343597383688 4-107374182467 5-369367187459 6-150323855412 7-171798691884 8-193273528359' 2024-09-17T16:09:55.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:55.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T16:09:55.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364829 2024-09-17T16:09:55.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364829 2024-09-17T16:09:55.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509520 3-343597383688 4-107374182467 5-369367187459 6-150323855412 7-171798691884 8-193273528359 9-214748364829' 2024-09-17T16:09:55.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:09:55.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-17T16:09:56.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201308 2024-09-17T16:09:56.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201308 2024-09-17T16:09:56.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509520 3-343597383688 4-107374182467 5-369367187459 6-150323855412 7-171798691884 8-193273528359 9-214748364829 10-236223201308' 2024-09-17T16:09:56.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:56.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836577 2024-09-17T16:09:56.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:56.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:09:56.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836577 2024-09-17T16:09:56.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:56.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836577 2024-09-17T16:09:56.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-09-17T16:09:56.107 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836577 2024-09-17T16:09:56.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:09:56.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836575 -lt 21474836577 2024-09-17T16:09:56.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:09:57.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:09:57.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:09:57.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836577 -lt 21474836577 2024-09-17T16:09:57.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:57.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673051 2024-09-17T16:09:57.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:57.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:09:57.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673051 2024-09-17T16:09:57.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:57.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673051 2024-09-17T16:09:57.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-09-17T16:09:57.934 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673051 2024-09-17T16:09:57.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:09:58.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673051 -lt 42949673051 2024-09-17T16:09:58.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:58.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509520 2024-09-17T16:09:58.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:58.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:09:58.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509520 2024-09-17T16:09:58.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:58.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509520 2024-09-17T16:09:58.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-09-17T16:09:58.357 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509520 2024-09-17T16:09:58.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:09:58.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509521 -lt 64424509520 2024-09-17T16:09:58.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:58.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-343597383688 2024-09-17T16:09:58.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:58.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:09:58.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-343597383688 2024-09-17T16:09:58.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:58.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383688 2024-09-17T16:09:58.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 343597383688' 2024-09-17T16:09:58.784 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 343597383688 2024-09-17T16:09:58.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:09:59.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383689 -lt 343597383688 2024-09-17T16:09:59.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:59.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182467 2024-09-17T16:09:59.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:59.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:09:59.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182467 2024-09-17T16:09:59.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:59.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182467 2024-09-17T16:09:59.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182467' 2024-09-17T16:09:59.211 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182467 2024-09-17T16:09:59.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:09:59.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182468 -lt 107374182467 2024-09-17T16:09:59.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:09:59.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-369367187459 2024-09-17T16:09:59.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:09:59.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:09:59.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-369367187459 2024-09-17T16:09:59.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:09:59.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=369367187459 2024-09-17T16:09:59.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 369367187459' 2024-09-17T16:09:59.600 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 369367187459 2024-09-17T16:09:59.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:10:00.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 369367187460 -lt 369367187459 2024-09-17T16:10:00.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:00.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855412 2024-09-17T16:10:00.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:00.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:10:00.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855412 2024-09-17T16:10:00.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:00.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855412 2024-09-17T16:10:00.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855412' 2024-09-17T16:10:00.030 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 150323855412 2024-09-17T16:10:00.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:10:00.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855412 -lt 150323855412 2024-09-17T16:10:00.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:00.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691884 2024-09-17T16:10:00.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:00.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T16:10:00.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691884 2024-09-17T16:10:00.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:00.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691884 2024-09-17T16:10:00.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691884' 2024-09-17T16:10:00.457 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691884 2024-09-17T16:10:00.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T16:10:00.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691885 -lt 171798691884 2024-09-17T16:10:00.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:00.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528359 2024-09-17T16:10:00.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:00.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T16:10:00.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528359 2024-09-17T16:10:00.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:00.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528359 2024-09-17T16:10:00.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528359' 2024-09-17T16:10:00.899 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528359 2024-09-17T16:10:00.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T16:10:01.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528360 -lt 193273528359 2024-09-17T16:10:01.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:01.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364829 2024-09-17T16:10:01.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:01.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T16:10:01.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364829 2024-09-17T16:10:01.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:01.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364829 2024-09-17T16:10:01.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364829' 2024-09-17T16:10:01.329 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 214748364829 2024-09-17T16:10:01.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T16:10:01.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364830 -lt 214748364829 2024-09-17T16:10:01.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:01.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201308 2024-09-17T16:10:01.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:01.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-17T16:10:01.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201308 2024-09-17T16:10:01.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:01.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201308 2024-09-17T16:10:01.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201308' 2024-09-17T16:10:01.762 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.10 seq 236223201308 2024-09-17T16:10:01.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-17T16:10:02.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201309 -lt 236223201308 2024-09-17T16:10:02.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:10:02.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:10:02.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:10:02.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-17T16:10:02.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:10:02.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:10:02.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:10:02.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:10:02.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:10:02.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:10:02.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:10:03.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-17T16:10:03.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:10:03.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:10:03.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:10:03.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-17T16:10:03.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:10:03.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:10:03.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:313: TEST_chunk_mapping: ceph osd erasure-code-profile set remap-profile plugin=lrc 'layers=[ [ "cDD", "" ] ]' mapping=_DD 'crush-steps=[ [ "choose", "osd", 0 ] ]' 2024-09-17T16:10:04.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:318: TEST_chunk_mapping: ceph osd erasure-code-profile get remap-profile 2024-09-17T16:10:04.693 INFO:tasks.workunit.client.0.smithi162.stdout:crush-device-class= 2024-09-17T16:10:04.693 INFO:tasks.workunit.client.0.smithi162.stdout:crush-failure-domain=host 2024-09-17T16:10:04.693 INFO:tasks.workunit.client.0.smithi162.stdout:crush-num-failure-domains=0 2024-09-17T16:10:04.693 INFO:tasks.workunit.client.0.smithi162.stdout:crush-osds-per-failure-domain=0 2024-09-17T16:10:04.693 INFO:tasks.workunit.client.0.smithi162.stdout:crush-root=default 2024-09-17T16:10:04.693 INFO:tasks.workunit.client.0.smithi162.stdout:crush-steps=[ [ "choose", "osd", 0 ] ] 2024-09-17T16:10:04.693 INFO:tasks.workunit.client.0.smithi162.stdout:k=-1 2024-09-17T16:10:04.693 INFO:tasks.workunit.client.0.smithi162.stdout:l=-1 2024-09-17T16:10:04.693 INFO:tasks.workunit.client.0.smithi162.stdout:layers=[ [ "cDD", "" ] ] 2024-09-17T16:10:04.693 INFO:tasks.workunit.client.0.smithi162.stdout:m=-1 2024-09-17T16:10:04.693 INFO:tasks.workunit.client.0.smithi162.stdout:mapping=_DD 2024-09-17T16:10:04.693 INFO:tasks.workunit.client.0.smithi162.stdout:plugin=lrc 2024-09-17T16:10:04.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:319: TEST_chunk_mapping: create_pool remap-pool 12 12 erasure remap-profile 2024-09-17T16:10:04.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create remap-pool 12 12 erasure remap-profile 2024-09-17T16:10:05.798 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'remap-pool' already exists 2024-09-17T16:10:05.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:10:06.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:327: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code remap-pool 1 2 2024-09-17T16:10:06.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-09-17T16:10:06.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=remap-pool 2024-09-17T16:10:06.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=1 2024-09-17T16:10:06.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=2 2024-09-17T16:10:06.813 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-17T16:10:06.813 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-17T16:10:06.832 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-17T16:10:06.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTremap-pool 2024-09-17T16:10:06.833 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-17T16:10:06.833 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-17T16:10:06.851 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-17T16:10:06.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDremap-pool 2024-09-17T16:10:06.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTremap-pool SECONDremap-pool' 2024-09-17T16:10:06.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTremap-pool SECONDremap-pool' 2024-09-17T16:10:06.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool remap-pool put SOMETHINGremap-pool td/test-erasure-code/ORIGINAL 2024-09-17T16:10:08.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool remap-pool get SOMETHINGremap-pool td/test-erasure-code/COPY 2024-09-17T16:10:08.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-09-17T16:10:08.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=remap-pool 2024-09-17T16:10:08.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGremap-pool 2024-09-17T16:10:08.915 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-09-17T16:10:08.915 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:10:09.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2024-09-17T16:10:09.329 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:10:09.329 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-09-17T16:10:09.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 6 9 2024-09-17T16:10:09.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('4' '6' '9') 2024-09-17T16:10:09.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-09-17T16:10:09.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-09-17T16:10:09.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-17T16:10:09.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.4 flush_journal 2024-09-17T16:10:09.421 INFO:tasks.workunit.client.0.smithi162.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-17T16:10:09.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-17T16:10:09.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-17T16:10:09.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.6 flush_journal 2024-09-17T16:10:09.519 INFO:tasks.workunit.client.0.smithi162.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-17T16:10:09.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-17T16:10:09.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-17T16:10:09.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-09-17T16:10:09.616 INFO:tasks.workunit.client.0.smithi162.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-17T16:10:09.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-17T16:10:09.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-17T16:10:09.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:10:09.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-09-17T16:10:09.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-09-17T16:10:09.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=remap-pool 2024-09-17T16:10:09.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGremap-pool 2024-09-17T16:10:09.620 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-09-17T16:10:09.620 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:10:10.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2024-09-17T16:10:10.049 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:10:10.049 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-09-17T16:10:10.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 6 9 2024-09-17T16:10:10.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('4' '6' '9') 2024-09-17T16:10:10.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-09-17T16:10:10.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-17T16:10:10.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2024-09-17T16:10:10.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:10:10.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=6 2024-09-17T16:10:10.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:10:10.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTremap-pool 2024-09-17T16:10:10.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-17T16:10:10.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-17T16:10:10.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:10:10.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=6 2024-09-17T16:10:10.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:10:10.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.6 2024-09-17T16:10:10.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:10:10.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:10:10.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:10:10.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:10:10.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:10:10.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:10:10.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-17T16:10:10.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-17T16:10:10.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:10:10.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=6 2024-09-17T16:10:10.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:10:10.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/6 2024-09-17T16:10:10.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/6 SOMETHINGremap-pool get-bytes 2024-09-17T16:10:11.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 6 2024-09-17T16:10:11.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2024-09-17T16:10:11.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:10:11.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=6 2024-09-17T16:10:11.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:10:11.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/6 2024-09-17T16:10:11.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:10:11.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:10:11.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:10:11.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:10:11.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-09-17T16:10:11.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-09-17T16:10:11.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:10:11.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:10:11.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-17T16:10:11.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:10:11.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:10:11.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:10:11.214 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:10:11.214 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:10:11.214 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:10:11.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:10:11.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:10:11.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:10:11.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-17T16:10:11.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-17T16:10:11.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:10:11.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:10:11.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:10:11.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:10:11.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:10:11.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:10:11.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/6 2024-09-17T16:10:11.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.6 2024-09-17T16:10:11.215 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.6 2024-09-17T16:10:11.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 6 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:10:11.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/6/whoami 2024-09-17T16:10:11.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 6 = 6 ']' 2024-09-17T16:10:11.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:10:11.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:10:11.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:10:11.239 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:10:11.238+0000 7f606adcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:10:11.287 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:10:11.286+0000 7f606adcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:10:11.289 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:10:11.287+0000 7f606adcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:10:11.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 6 2024-09-17T16:10:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:10:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-17T16:10:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:10:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:10:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:10:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:10:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:10:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:10:11.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:10:12.141 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:10:12.140+0000 7f606adcc640 -1 Falling back to public interface 2024-09-17T16:10:12.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:10:13.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:10:13.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:10:13.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:10:13.349 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:10:13.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:10:13.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:10:13.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:10:14.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:10:14.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:10:14.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:10:14.791 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:10:14.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:10:14.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:10:15.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:10:15.977 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:10:15.974+0000 7f606adcc640 -1 osd.6 96 log_to_monitors true 2024-09-17T16:10:16.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:10:16.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:10:16.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:10:16.345 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:10:16.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:10:16.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:10:16.879 INFO:tasks.workunit.client.0.smithi162.stderr:osd.6 up in weight 1 up_from 103 up_thru 92 down_at 99 last_clean_interval [35,96) [v2:127.0.0.1:6850/1234850421,v1:127.0.0.1:6851/1234850421] [v2:127.0.0.1:6852/1234850421,v1:127.0.0.1:6853/1234850421] exists,up 847dfb63-8a98-445f-976b-8f106467502e 2024-09-17T16:10:16.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:10:16.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:10:16.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:10:16.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:10:16.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:10:16.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:10:16.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:10:16.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:10:16.880 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:10:16.880 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:10:16.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:10:16.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:10:16.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:10:16.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:10:16.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:10:16.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:10:16.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:10:16.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:10:16.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:10:17.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:10:17.600 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:10:17.600 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:10:17.600 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:10:17.600 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:10:17.600 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:10:17.600 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:10:17.600 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:10:17.600 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-17T16:10:17.600 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:10:17.601 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-17T16:10:17.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:10:17.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:17.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:10:17.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836583 2024-09-17T16:10:17.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836583 2024-09-17T16:10:17.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583' 2024-09-17T16:10:17.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:17.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:10:18.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673057 2024-09-17T16:10:18.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673057 2024-09-17T16:10:18.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057' 2024-09-17T16:10:18.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:18.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:10:18.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509527 2024-09-17T16:10:18.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509527 2024-09-17T16:10:18.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509527' 2024-09-17T16:10:18.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:18.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:10:18.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383695 2024-09-17T16:10:18.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383695 2024-09-17T16:10:18.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509527 3-343597383695' 2024-09-17T16:10:18.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:18.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:10:18.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182474 2024-09-17T16:10:18.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182474 2024-09-17T16:10:18.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509527 3-343597383695 4-107374182474' 2024-09-17T16:10:18.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:18.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:10:18.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=369367187466 2024-09-17T16:10:18.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 369367187466 2024-09-17T16:10:18.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509527 3-343597383695 4-107374182474 5-369367187466' 2024-09-17T16:10:18.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:18.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:10:19.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=442381631491 2024-09-17T16:10:19.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 442381631491 2024-09-17T16:10:19.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509527 3-343597383695 4-107374182474 5-369367187466 6-442381631491' 2024-09-17T16:10:19.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:19.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T16:10:19.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691891 2024-09-17T16:10:19.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691891 2024-09-17T16:10:19.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509527 3-343597383695 4-107374182474 5-369367187466 6-442381631491 7-171798691891' 2024-09-17T16:10:19.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:19.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T16:10:19.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528366 2024-09-17T16:10:19.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528366 2024-09-17T16:10:19.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509527 3-343597383695 4-107374182474 5-369367187466 6-442381631491 7-171798691891 8-193273528366' 2024-09-17T16:10:19.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:19.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T16:10:19.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364836 2024-09-17T16:10:19.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364836 2024-09-17T16:10:19.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509527 3-343597383695 4-107374182474 5-369367187466 6-442381631491 7-171798691891 8-193273528366 9-214748364836' 2024-09-17T16:10:19.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:19.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-17T16:10:19.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201315 2024-09-17T16:10:19.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201315 2024-09-17T16:10:19.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509527 3-343597383695 4-107374182474 5-369367187466 6-442381631491 7-171798691891 8-193273528366 9-214748364836 10-236223201315' 2024-09-17T16:10:19.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:19.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836583 2024-09-17T16:10:19.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:19.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:10:19.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836583 2024-09-17T16:10:19.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:19.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836583 2024-09-17T16:10:19.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836583' 2024-09-17T16:10:19.577 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836583 2024-09-17T16:10:19.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:10:20.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836583 -lt 21474836583 2024-09-17T16:10:20.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:20.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673057 2024-09-17T16:10:20.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:20.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:10:20.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673057 2024-09-17T16:10:20.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:20.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673057 2024-09-17T16:10:20.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-09-17T16:10:20.005 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673057 2024-09-17T16:10:20.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:10:20.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673058 -lt 42949673057 2024-09-17T16:10:20.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:20.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509527 2024-09-17T16:10:20.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:20.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:10:20.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509527 2024-09-17T16:10:20.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:20.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509527 2024-09-17T16:10:20.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2024-09-17T16:10:20.473 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509527 2024-09-17T16:10:20.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:10:20.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509527 -lt 64424509527 2024-09-17T16:10:20.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:20.958 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-343597383695 2024-09-17T16:10:20.958 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:20.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:10:20.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-343597383695 2024-09-17T16:10:20.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:20.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383695 2024-09-17T16:10:20.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 343597383695' 2024-09-17T16:10:20.961 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 343597383695 2024-09-17T16:10:20.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:10:21.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383695 -lt 343597383695 2024-09-17T16:10:21.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:21.386 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182474 2024-09-17T16:10:21.386 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:10:21.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182474 2024-09-17T16:10:21.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:21.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182474 2024-09-17T16:10:21.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-09-17T16:10:21.388 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182474 2024-09-17T16:10:21.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:10:21.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182474 -lt 107374182474 2024-09-17T16:10:21.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:21.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-369367187466 2024-09-17T16:10:21.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:21.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:10:21.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-369367187466 2024-09-17T16:10:21.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:21.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=369367187466 2024-09-17T16:10:21.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 369367187466' 2024-09-17T16:10:21.815 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 369367187466 2024-09-17T16:10:21.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:10:22.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 369367187466 -lt 369367187466 2024-09-17T16:10:22.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:22.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-442381631491 2024-09-17T16:10:22.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:22.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:10:22.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-442381631491 2024-09-17T16:10:22.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=442381631491 2024-09-17T16:10:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 442381631491' 2024-09-17T16:10:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 442381631491 2024-09-17T16:10:22.258 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:10:22.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 442381631491 -lt 442381631491 2024-09-17T16:10:22.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:22.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691891 2024-09-17T16:10:22.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:22.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T16:10:22.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691891 2024-09-17T16:10:22.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:22.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691891 2024-09-17T16:10:22.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691891' 2024-09-17T16:10:22.685 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691891 2024-09-17T16:10:22.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T16:10:23.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691892 -lt 171798691891 2024-09-17T16:10:23.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:23.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528366 2024-09-17T16:10:23.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:23.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T16:10:23.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528366 2024-09-17T16:10:23.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:23.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528366 2024-09-17T16:10:23.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528366' 2024-09-17T16:10:23.075 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528366 2024-09-17T16:10:23.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T16:10:23.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528367 -lt 193273528366 2024-09-17T16:10:23.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:23.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364836 2024-09-17T16:10:23.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:23.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T16:10:23.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364836 2024-09-17T16:10:23.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:23.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364836 2024-09-17T16:10:23.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364836' 2024-09-17T16:10:23.476 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 214748364836 2024-09-17T16:10:23.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T16:10:23.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364836 -lt 214748364836 2024-09-17T16:10:23.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:23.866 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201315 2024-09-17T16:10:23.866 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:23.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-17T16:10:23.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201315 2024-09-17T16:10:23.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:23.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201315 2024-09-17T16:10:23.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201315' 2024-09-17T16:10:23.868 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.10 seq 236223201315 2024-09-17T16:10:23.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-17T16:10:24.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201316 -lt 236223201315 2024-09-17T16:10:24.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:10:24.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:10:24.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:10:24.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 28 == 0 2024-09-17T16:10:24.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:10:24.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:10:24.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:10:24.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:10:24.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:10:24.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:10:24.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:10:25.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=28 2024-09-17T16:10:25.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:10:25.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:10:25.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:10:25.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 = 28 2024-09-17T16:10:25.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:10:25.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:10:25.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-17T16:10:25.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2024-09-17T16:10:25.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:10:25.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDremap-pool 2024-09-17T16:10:25.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-17T16:10:25.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:10:25.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-17T16:10:25.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-17T16:10:25.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:10:25.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-17T16:10:25.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:10:25.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.9 2024-09-17T16:10:25.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:10:25.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:10:25.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:10:25.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:10:25.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:10:26.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:10:26.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-17T16:10:26.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-17T16:10:26.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:10:26.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-17T16:10:26.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:10:26.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/9 2024-09-17T16:10:26.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/9 SOMETHINGremap-pool get-bytes 2024-09-17T16:10:26.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 9 2024-09-17T16:10:26.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2024-09-17T16:10:26.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:10:26.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-17T16:10:26.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:10:26.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/9 2024-09-17T16:10:26.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-17T16:10:26.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:10:26.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:10:26.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:10:26.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-09-17T16:10:26.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-09-17T16:10:26.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:10:26.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:10:26.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-17T16:10:26.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:10:26.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:10:26.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:10:26.974 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:10:26.974 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:10:26.975 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:10:26.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:10:26.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' 2024-09-17T16:10:26.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:10:26.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-17T16:10:26.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-17T16:10:26.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:10:26.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:10:26.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:10:26.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:10:26.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:10:26.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:10:26.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/9 2024-09-17T16:10:26.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-17T16:10:26.975 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.9 2024-09-17T16:10:26.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=e3019c81-bf67-43d8-9e68-76269b2f84e1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.56062/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:10:26.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/9/whoami 2024-09-17T16:10:26.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-17T16:10:26.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:10:26.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:10:26.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:10:27.002 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:10:27.001+0000 7f6b83e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:10:27.005 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:10:27.003+0000 7f6b83e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:10:27.006 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:10:27.004+0000 7f6b83e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:10:27.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-17T16:10:27.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:10:27.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-17T16:10:27.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:10:27.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:10:27.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:10:27.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:10:27.716 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:10:27.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:10:27.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:10:28.115 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:10:28.113+0000 7f6b83e14640 -1 Falling back to public interface 2024-09-17T16:10:28.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:10:29.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:10:29.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:10:29.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:10:29.278 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:10:29.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:10:29.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:10:29.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:10:30.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:10:30.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:10:30.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:10:30.708 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:10:30.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:10:30.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:10:31.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:10:31.412 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:10:31.410+0000 7f6b83e14640 -1 osd.9 106 log_to_monitors true 2024-09-17T16:10:32.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:10:32.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:10:32.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:10:32.167 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:10:32.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:10:32.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T16:10:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:osd.9 up in weight 1 up_from 111 up_thru 69 down_at 107 last_clean_interval [50,106) [v2:127.0.0.1:6874/292097971,v1:127.0.0.1:6875/292097971] [v2:127.0.0.1:6876/292097971,v1:127.0.0.1:6877/292097971] exists,up b2db456f-7e44-4dbd-85f8-de0b48f51a42 2024-09-17T16:10:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:10:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:10:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:10:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:10:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:10:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:10:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:10:33.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:10:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:10:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:10:33.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:10:33.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:10:33.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:10:33.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:10:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:10:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:10:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:10:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:10:33.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:10:34.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:10:34.052 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:10:34.052 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:10:34.053 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:10:34.053 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:10:34.053 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:10:34.053 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:10:34.053 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:10:34.053 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-17T16:10:34.053 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:10:34.053 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-17T16:10:34.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:10:34.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:34.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:10:34.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836589 2024-09-17T16:10:34.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836589 2024-09-17T16:10:34.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589' 2024-09-17T16:10:34.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:34.167 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:10:34.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673063 2024-09-17T16:10:34.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673063 2024-09-17T16:10:34.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063' 2024-09-17T16:10:34.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:34.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:10:34.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509532 2024-09-17T16:10:34.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509532 2024-09-17T16:10:34.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532' 2024-09-17T16:10:34.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:34.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:10:34.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383700 2024-09-17T16:10:34.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383700 2024-09-17T16:10:34.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532 3-343597383700' 2024-09-17T16:10:34.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:34.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:10:34.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182479 2024-09-17T16:10:34.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182479 2024-09-17T16:10:34.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532 3-343597383700 4-107374182479' 2024-09-17T16:10:34.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:34.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:10:35.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=369367187471 2024-09-17T16:10:35.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 369367187471 2024-09-17T16:10:35.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532 3-343597383700 4-107374182479 5-369367187471' 2024-09-17T16:10:35.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:35.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:10:35.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=442381631496 2024-09-17T16:10:35.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 442381631496 2024-09-17T16:10:35.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532 3-343597383700 4-107374182479 5-369367187471 6-442381631496' 2024-09-17T16:10:35.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:35.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T16:10:35.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691896 2024-09-17T16:10:35.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691896 2024-09-17T16:10:35.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532 3-343597383700 4-107374182479 5-369367187471 6-442381631496 7-171798691896' 2024-09-17T16:10:35.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:35.258 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T16:10:35.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528371 2024-09-17T16:10:35.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528371 2024-09-17T16:10:35.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532 3-343597383700 4-107374182479 5-369367187471 6-442381631496 7-171798691896 8-193273528371' 2024-09-17T16:10:35.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:35.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T16:10:35.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=476741369859 2024-09-17T16:10:35.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 476741369859 2024-09-17T16:10:35.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532 3-343597383700 4-107374182479 5-369367187471 6-442381631496 7-171798691896 8-193273528371 9-476741369859' 2024-09-17T16:10:35.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:10:35.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-17T16:10:35.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201320 2024-09-17T16:10:35.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201320 2024-09-17T16:10:35.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532 3-343597383700 4-107374182479 5-369367187471 6-442381631496 7-171798691896 8-193273528371 9-476741369859 10-236223201320' 2024-09-17T16:10:35.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:35.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836589 2024-09-17T16:10:35.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:35.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:10:35.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836589 2024-09-17T16:10:35.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:35.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836589 2024-09-17T16:10:35.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-09-17T16:10:35.596 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836589 2024-09-17T16:10:35.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:10:36.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836589 -lt 21474836589 2024-09-17T16:10:36.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:36.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673063 2024-09-17T16:10:36.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:36.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:10:36.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673063 2024-09-17T16:10:36.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:36.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673063 2024-09-17T16:10:36.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2024-09-17T16:10:36.025 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673063 2024-09-17T16:10:36.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:10:36.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673063 -lt 42949673063 2024-09-17T16:10:36.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:36.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509532 2024-09-17T16:10:36.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:36.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:10:36.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509532 2024-09-17T16:10:36.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:36.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509532 2024-09-17T16:10:36.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-09-17T16:10:36.452 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509532 2024-09-17T16:10:36.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:10:36.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509532 -lt 64424509532 2024-09-17T16:10:36.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:36.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-343597383700 2024-09-17T16:10:36.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:36.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:10:36.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-343597383700 2024-09-17T16:10:36.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:36.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383700 2024-09-17T16:10:36.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 343597383700' 2024-09-17T16:10:36.840 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 343597383700 2024-09-17T16:10:36.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:10:37.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383701 -lt 343597383700 2024-09-17T16:10:37.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:37.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182479 2024-09-17T16:10:37.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:37.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:10:37.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182479 2024-09-17T16:10:37.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:37.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182479 2024-09-17T16:10:37.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182479' 2024-09-17T16:10:37.605 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182479 2024-09-17T16:10:37.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:10:38.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182480 -lt 107374182479 2024-09-17T16:10:38.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:38.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-369367187471 2024-09-17T16:10:38.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:38.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:10:38.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-369367187471 2024-09-17T16:10:38.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:38.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=369367187471 2024-09-17T16:10:38.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 369367187471' 2024-09-17T16:10:38.040 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 369367187471 2024-09-17T16:10:38.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:10:38.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 369367187471 -lt 369367187471 2024-09-17T16:10:38.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:38.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-442381631496 2024-09-17T16:10:38.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:38.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:10:38.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-442381631496 2024-09-17T16:10:38.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:38.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=442381631496 2024-09-17T16:10:38.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 442381631496' 2024-09-17T16:10:38.466 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 442381631496 2024-09-17T16:10:38.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:10:38.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 442381631497 -lt 442381631496 2024-09-17T16:10:38.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:38.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691896 2024-09-17T16:10:38.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:38.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T16:10:38.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691896 2024-09-17T16:10:38.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:38.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691896 2024-09-17T16:10:38.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691896' 2024-09-17T16:10:38.917 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691896 2024-09-17T16:10:38.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T16:10:39.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691897 -lt 171798691896 2024-09-17T16:10:39.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:39.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528371 2024-09-17T16:10:39.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:39.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T16:10:39.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528371 2024-09-17T16:10:39.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:39.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528371 2024-09-17T16:10:39.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528371' 2024-09-17T16:10:39.378 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528371 2024-09-17T16:10:39.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T16:10:39.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528372 -lt 193273528371 2024-09-17T16:10:39.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:39.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-476741369859 2024-09-17T16:10:39.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:39.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T16:10:39.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-476741369859 2024-09-17T16:10:39.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:39.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=476741369859 2024-09-17T16:10:39.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 476741369859' 2024-09-17T16:10:39.788 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 476741369859 2024-09-17T16:10:39.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T16:10:40.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 476741369860 -lt 476741369859 2024-09-17T16:10:40.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:10:40.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201320 2024-09-17T16:10:40.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:10:40.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-17T16:10:40.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201320 2024-09-17T16:10:40.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:10:40.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201320 2024-09-17T16:10:40.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201320' 2024-09-17T16:10:40.214 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.10 seq 236223201320 2024-09-17T16:10:40.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-17T16:10:40.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201321 -lt 236223201320 2024-09-17T16:10:40.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:10:40.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:10:40.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:10:41.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 28 == 0 2024-09-17T16:10:41.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:10:41.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:10:41.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:10:41.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:10:41.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:10:41.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:10:41.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:10:41.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=28 2024-09-17T16:10:41.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:10:41.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:10:41.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:10:42.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 = 28 2024-09-17T16:10:42.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:10:42.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:10:42.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:329: TEST_chunk_mapping: delete_pool remap-pool 2024-09-17T16:10:42.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=remap-pool 2024-09-17T16:10:42.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete remap-pool remap-pool --yes-i-really-really-mean-it 2024-09-17T16:10:42.667 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'remap-pool' does not exist 2024-09-17T16:10:42.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:330: TEST_chunk_mapping: ceph osd erasure-code-profile rm remap-profile 2024-09-17T16:10:43.892 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile remap-profile does not exist 2024-09-17T16:10:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-17T16:10:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_isa td/test-erasure-code 2024-09-17T16:10:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:193: TEST_rados_put_get_isa: erasure_code_plugin_exists isa 2024-09-17T16:10:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: erasure_code_plugin_exists: local plugin=isa 2024-09-17T16:10:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: local status 2024-09-17T16:10:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: erasure_code_plugin_exists: local grepstr 2024-09-17T16:10:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: erasure_code_plugin_exists: local s 2024-09-17T16:10:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: case `uname` in 2024-09-17T16:10:43.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: uname 2024-09-17T16:10:43.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: erasure_code_plugin_exists: grepstr='isa.*No such file' 2024-09-17T16:10:43.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=isa 2024-09-17T16:10:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: s= 2024-09-17T16:10:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: local status=0 2024-09-17T16:10:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-09-17T16:10:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-09-17T16:10:45.368 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile TESTPROFILE does not exist 2024-09-17T16:10:45.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: erasure_code_plugin_exists: return 0 2024-09-17T16:10:45.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:197: TEST_rados_put_get_isa: local dir=td/test-erasure-code 2024-09-17T16:10:45.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:198: TEST_rados_put_get_isa: local poolname=pool-isa 2024-09-17T16:10:45.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:200: TEST_rados_put_get_isa: ceph osd erasure-code-profile set profile-isa plugin=isa crush-failure-domain=osd 2024-09-17T16:10:45.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:203: TEST_rados_put_get_isa: create_pool pool-isa 1 1 erasure profile-isa 2024-09-17T16:10:45.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-isa 1 1 erasure profile-isa 2024-09-17T16:10:46.737 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-isa' already exists 2024-09-17T16:10:46.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:10:47.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:206: TEST_rados_put_get_isa: rados_put_get td/test-erasure-code pool-isa 2024-09-17T16:10:47.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-17T16:10:47.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-isa 2024-09-17T16:10:47.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-17T16:10:47.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:10:47.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-17T16:10:47.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:10:47.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-17T16:10:47.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:10:47.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-17T16:10:47.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:10:47.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-17T16:10:47.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-isa put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-17T16:10:49.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-09-17T16:10:49.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:10:49.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-17T16:10:49.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-isa SOMETHING 2024-09-17T16:10:49.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-isa 2024-09-17T16:10:49.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:10:49.481 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-09-17T16:10:49.481 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:10:49.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2024-09-17T16:10:49.908 INFO:tasks.workunit.client.0.smithi162.stderr:10 2024-09-17T16:10:49.908 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:10:49.908 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:10:49.908 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:10:49.908 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-17T16:10:49.908 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:10:49.908 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:10:49.908 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:10:49.909 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-09-17T16:10:49.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 10 0 7 6 8 2 1 5 9 2024-09-17T16:10:49.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('4' '10' '0' '7' '6' '8' '2' '1' '5' '9') 2024-09-17T16:10:49.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-17T16:10:49.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=9 2024-09-17T16:10:49.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-09-17T16:10:50.400 INFO:tasks.workunit.client.0.smithi162.stderr:osd.9 is already out. 2024-09-17T16:10:50.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-17T16:10:55.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-isa SOMETHING 2024-09-17T16:10:55.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-isa 2024-09-17T16:10:55.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:10:55.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-09-17T16:10:55.415 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-09-17T16:10:55.415 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:10:55.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2024-09-17T16:10:55.811 INFO:tasks.workunit.client.0.smithi162.stderr:10 2024-09-17T16:10:55.811 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:10:55.811 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:10:55.811 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:10:55.811 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-17T16:10:55.811 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:10:55.811 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:10:55.812 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:10:55.812 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:10:55.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 10 0 7 6 8 3 1 5 2 2024-09-17T16:10:55.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-09-17T16:10:55.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:10:55.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-09-17T16:10:56.353 INFO:tasks.workunit.client.0.smithi162.stderr:osd.9 is already in. 2024-09-17T16:10:56.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-17T16:10:56.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:208: TEST_rados_put_get_isa: delete_pool pool-isa 2024-09-17T16:10:56.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-isa 2024-09-17T16:10:56.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-isa pool-isa --yes-i-really-really-mean-it 2024-09-17T16:10:58.168 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-isa' does not exist 2024-09-17T16:10:58.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-17T16:10:58.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_jerasure td/test-erasure-code 2024-09-17T16:10:58.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:212: TEST_rados_put_get_jerasure: local dir=td/test-erasure-code 2024-09-17T16:10:58.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:214: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code ecpool 2024-09-17T16:10:58.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-17T16:10:58.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=ecpool 2024-09-17T16:10:58.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-17T16:10:58.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:10:58.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-17T16:10:58.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:10:58.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-17T16:10:58.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:10:58.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-17T16:10:58.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:10:58.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-17T16:10:58.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool ecpool put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-17T16:10:58.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-09-17T16:10:59.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:10:59.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-17T16:10:59.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds ecpool SOMETHING 2024-09-17T16:10:59.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-17T16:10:59.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:10:59.061 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-17T16:10:59.061 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:10:59.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=6 2024-09-17T16:10:59.535 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:10:59.535 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:10:59.535 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-09-17T16:10:59.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 6 0 7 9 2024-09-17T16:10:59.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('6' '0' '7' '9') 2024-09-17T16:10:59.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-17T16:10:59.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2024-09-17T16:10:59.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-09-17T16:11:00.175 INFO:tasks.workunit.client.0.smithi162.stderr:osd.9 is already out. 2024-09-17T16:11:00.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-17T16:11:05.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds ecpool SOMETHING 2024-09-17T16:11:05.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-17T16:11:05.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-09-17T16:11:05.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:11:05.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-17T16:11:05.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:11:05.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=6 2024-09-17T16:11:05.623 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:11:05.623 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:11:05.623 INFO:tasks.workunit.client.0.smithi162.stderr:5' 2024-09-17T16:11:05.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 6 0 7 5 2024-09-17T16:11:05.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-09-17T16:11:05.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:11:05.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-09-17T16:11:06.185 INFO:tasks.workunit.client.0.smithi162.stderr:osd.9 is already in. 2024-09-17T16:11:06.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-17T16:11:06.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:216: TEST_rados_put_get_jerasure: local poolname=pool-jerasure 2024-09-17T16:11:06.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:217: TEST_rados_put_get_jerasure: local profile=profile-jerasure 2024-09-17T16:11:06.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:219: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile set profile-jerasure plugin=jerasure k=4 m=2 crush-failure-domain=osd 2024-09-17T16:11:07.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:223: TEST_rados_put_get_jerasure: create_pool pool-jerasure 12 12 erasure profile-jerasure 2024-09-17T16:11:07.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 12 12 erasure profile-jerasure 2024-09-17T16:11:10.176 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-17T16:11:10.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:11:11.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:226: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code pool-jerasure 2024-09-17T16:11:11.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-17T16:11:11.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-jerasure 2024-09-17T16:11:11.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-17T16:11:11.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:11:11.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-17T16:11:11.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:11:11.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-17T16:11:11.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:11:11.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-17T16:11:11.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:11:11.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-17T16:11:11.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-17T16:11:12.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-17T16:11:12.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:11:12.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-17T16:11:12.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-jerasure SOMETHING 2024-09-17T16:11:12.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:11:12.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:11:12.643 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-17T16:11:12.643 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:11:13.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-17T16:11:13.053 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:11:13.053 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:11:13.053 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:11:13.054 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:11:13.054 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-17T16:11:13.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 10 2024-09-17T16:11:13.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '1' '9' '4' '3' '10') 2024-09-17T16:11:13.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-17T16:11:13.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=5 2024-09-17T16:11:13.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 10 2024-09-17T16:11:13.911 INFO:tasks.workunit.client.0.smithi162.stderr:osd.10 is already out. 2024-09-17T16:11:13.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-17T16:11:18.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-jerasure SOMETHING 2024-09-17T16:11:18.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:11:18.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:11:18.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<10\>' 2024-09-17T16:11:18.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-17T16:11:18.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:11:19.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-17T16:11:19.349 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:11:19.349 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:11:19.349 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:11:19.349 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:11:19.349 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-17T16:11:19.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 6 2024-09-17T16:11:19.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-17T16:11:19.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:11:19.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 10 2024-09-17T16:11:19.949 INFO:tasks.workunit.client.0.smithi162.stderr:osd.10 is already in. 2024-09-17T16:11:19.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-17T16:11:19.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:227: TEST_rados_put_get_jerasure: rados_osds_out_in td/test-erasure-code pool-jerasure 2024-09-17T16:11:19.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:104: rados_osds_out_in: local dir=td/test-erasure-code 2024-09-17T16:11:19.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: local poolname=pool-jerasure 2024-09-17T16:11:19.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: local objname=SOMETHING 2024-09-17T16:11:19.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-17T16:11:19.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 FFFF 2024-09-17T16:11:19.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-17T16:11:19.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 GGGG 2024-09-17T16:11:19.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-17T16:11:19.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 HHHH 2024-09-17T16:11:19.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-17T16:11:19.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 IIII 2024-09-17T16:11:19.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:116: rados_osds_out_in: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-17T16:11:22.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:117: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-17T16:11:22.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:118: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:11:22.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:119: rados_osds_out_in: rm td/test-erasure-code/COPY 2024-09-17T16:11:22.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: wait_for_clean 2024-09-17T16:11:22.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:11:22.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:11:22.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:11:22.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:11:22.430 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:11:22.430 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:11:22.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:11:22.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:11:22.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:11:22.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:11:22.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:11:22.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:11:22.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:11:22.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:11:22.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:11:22.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:11:22.979 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:11:22.979 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:11:22.979 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:11:22.979 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:11:22.979 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:11:22.979 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:11:22.979 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:11:22.979 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-17T16:11:22.979 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:11:22.979 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-17T16:11:22.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:11:22.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:22.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:11:23.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836600 2024-09-17T16:11:23.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836600 2024-09-17T16:11:23.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600' 2024-09-17T16:11:23.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:23.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:11:23.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673074 2024-09-17T16:11:23.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673074 2024-09-17T16:11:23.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673074' 2024-09-17T16:11:23.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:23.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:11:23.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509544 2024-09-17T16:11:23.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509544 2024-09-17T16:11:23.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673074 2-64424509544' 2024-09-17T16:11:23.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:23.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:11:23.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383712 2024-09-17T16:11:23.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383712 2024-09-17T16:11:23.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673074 2-64424509544 3-343597383712' 2024-09-17T16:11:23.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:23.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:11:23.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182491 2024-09-17T16:11:23.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182491 2024-09-17T16:11:23.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673074 2-64424509544 3-343597383712 4-107374182491' 2024-09-17T16:11:23.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:23.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:11:23.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=369367187483 2024-09-17T16:11:23.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 369367187483 2024-09-17T16:11:23.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673074 2-64424509544 3-343597383712 4-107374182491 5-369367187483' 2024-09-17T16:11:23.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:23.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:11:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=442381631508 2024-09-17T16:11:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 442381631508 2024-09-17T16:11:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673074 2-64424509544 3-343597383712 4-107374182491 5-369367187483 6-442381631508' 2024-09-17T16:11:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:24.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T16:11:24.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691908 2024-09-17T16:11:24.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691908 2024-09-17T16:11:24.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673074 2-64424509544 3-343597383712 4-107374182491 5-369367187483 6-442381631508 7-171798691908' 2024-09-17T16:11:24.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:24.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T16:11:24.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528383 2024-09-17T16:11:24.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528383 2024-09-17T16:11:24.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673074 2-64424509544 3-343597383712 4-107374182491 5-369367187483 6-442381631508 7-171798691908 8-193273528383' 2024-09-17T16:11:24.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:24.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T16:11:24.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=476741369871 2024-09-17T16:11:24.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 476741369871 2024-09-17T16:11:24.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673074 2-64424509544 3-343597383712 4-107374182491 5-369367187483 6-442381631508 7-171798691908 8-193273528383 9-476741369871' 2024-09-17T16:11:24.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:24.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-17T16:11:24.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201332 2024-09-17T16:11:24.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201332 2024-09-17T16:11:24.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673074 2-64424509544 3-343597383712 4-107374182491 5-369367187483 6-442381631508 7-171798691908 8-193273528383 9-476741369871 10-236223201332' 2024-09-17T16:11:24.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:24.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836600 2024-09-17T16:11:24.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:24.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:11:24.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836600 2024-09-17T16:11:24.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:24.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836600 2024-09-17T16:11:24.567 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836600 2024-09-17T16:11:24.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836600' 2024-09-17T16:11:24.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:11:24.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836601 -lt 21474836600 2024-09-17T16:11:24.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:24.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673074 2024-09-17T16:11:24.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:24.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:11:24.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673074 2024-09-17T16:11:24.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:24.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673074 2024-09-17T16:11:24.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673074' 2024-09-17T16:11:24.967 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949673074 2024-09-17T16:11:24.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:11:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673075 -lt 42949673074 2024-09-17T16:11:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:25.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509544 2024-09-17T16:11:25.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:25.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:11:25.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509544 2024-09-17T16:11:25.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:25.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509544 2024-09-17T16:11:25.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509544' 2024-09-17T16:11:25.362 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509544 2024-09-17T16:11:25.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:11:25.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509544 -lt 64424509544 2024-09-17T16:11:25.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:25.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-343597383712 2024-09-17T16:11:25.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:25.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:11:25.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-343597383712 2024-09-17T16:11:25.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383712 2024-09-17T16:11:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 343597383712' 2024-09-17T16:11:25.831 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 343597383712 2024-09-17T16:11:25.831 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:11:26.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383712 -lt 343597383712 2024-09-17T16:11:26.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:26.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182491 2024-09-17T16:11:26.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:26.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:11:26.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182491 2024-09-17T16:11:26.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:26.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182491 2024-09-17T16:11:26.257 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182491 2024-09-17T16:11:26.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182491' 2024-09-17T16:11:26.258 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:11:26.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182491 -lt 107374182491 2024-09-17T16:11:26.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:26.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-369367187483 2024-09-17T16:11:26.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:26.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:11:26.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-369367187483 2024-09-17T16:11:26.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:26.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=369367187483 2024-09-17T16:11:26.684 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 369367187483 2024-09-17T16:11:26.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 369367187483' 2024-09-17T16:11:26.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:11:27.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 369367187483 -lt 369367187483 2024-09-17T16:11:27.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:27.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-442381631508 2024-09-17T16:11:27.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:27.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:11:27.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-442381631508 2024-09-17T16:11:27.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:27.119 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 442381631508 2024-09-17T16:11:27.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=442381631508 2024-09-17T16:11:27.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 442381631508' 2024-09-17T16:11:27.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:11:27.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 442381631509 -lt 442381631508 2024-09-17T16:11:27.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:27.543 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691908 2024-09-17T16:11:27.543 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:27.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T16:11:27.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691908 2024-09-17T16:11:27.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:27.546 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691908 2024-09-17T16:11:27.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691908 2024-09-17T16:11:27.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691908' 2024-09-17T16:11:27.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T16:11:27.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691909 -lt 171798691908 2024-09-17T16:11:27.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:27.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528383 2024-09-17T16:11:27.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:27.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T16:11:27.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528383 2024-09-17T16:11:27.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:27.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528383 2024-09-17T16:11:27.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528383' 2024-09-17T16:11:27.973 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 193273528383 2024-09-17T16:11:27.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T16:11:28.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528383 -lt 193273528383 2024-09-17T16:11:28.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:28.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-476741369871 2024-09-17T16:11:28.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:28.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T16:11:28.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-476741369871 2024-09-17T16:11:28.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:28.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=476741369871 2024-09-17T16:11:28.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 476741369871' 2024-09-17T16:11:28.362 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.9 seq 476741369871 2024-09-17T16:11:28.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T16:11:28.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 476741369872 -lt 476741369871 2024-09-17T16:11:28.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:28.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201332 2024-09-17T16:11:28.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:28.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-17T16:11:28.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201332 2024-09-17T16:11:28.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:28.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201332 2024-09-17T16:11:28.786 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.10 seq 236223201332 2024-09-17T16:11:28.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201332' 2024-09-17T16:11:28.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-17T16:11:29.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201333 -lt 236223201332 2024-09-17T16:11:29.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:11:29.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:11:29.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:11:29.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 28 == 0 2024-09-17T16:11:29.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:11:29.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:11:29.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:11:29.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:11:29.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:11:29.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:11:29.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:11:30.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=28 2024-09-17T16:11:30.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:11:30.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:11:30.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:11:30.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 = 28 2024-09-17T16:11:30.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:11:30.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:11:30.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-17T16:11:30.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:11:30.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:11:30.657 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-17T16:11:30.657 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:11:31.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-17T16:11:31.125 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:11:31.125 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:11:31.125 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:11:31.125 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:11:31.125 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-17T16:11:31.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 10 2024-09-17T16:11:31.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: local 'osds_list=5 1 9 4 3 10' 2024-09-17T16:11:31.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: osds=('5' '1' '9' '4' '3' '10') 2024-09-17T16:11:31.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: local -a osds 2024-09-17T16:11:31.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-09-17T16:11:31.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 5 2024-09-17T16:11:31.637 INFO:tasks.workunit.client.0.smithi162.stderr:osd.5 is already out. 2024-09-17T16:11:31.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-09-17T16:11:31.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 1 2024-09-17T16:11:33.537 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 is already out. 2024-09-17T16:11:33.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: wait_for_clean 2024-09-17T16:11:33.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:11:33.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:11:33.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:11:33.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:11:33.550 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:11:33.551 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:11:33.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:11:33.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:11:33.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:11:33.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:11:33.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:11:33.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:11:33.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:11:33.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:11:33.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:11:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:11:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:11:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:11:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:11:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:11:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:11:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:11:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:11:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-17T16:11:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:11:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-17T16:11:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:11:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:34.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:11:34.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836605 2024-09-17T16:11:34.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836605 2024-09-17T16:11:34.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836605' 2024-09-17T16:11:34.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:34.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:11:34.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673079 2024-09-17T16:11:34.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673079 2024-09-17T16:11:34.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836605 1-42949673079' 2024-09-17T16:11:34.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:34.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:11:35.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509548 2024-09-17T16:11:35.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509548 2024-09-17T16:11:35.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509548' 2024-09-17T16:11:35.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:35.156 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:11:35.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383716 2024-09-17T16:11:35.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383716 2024-09-17T16:11:35.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509548 3-343597383716' 2024-09-17T16:11:35.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:35.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:11:35.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182495 2024-09-17T16:11:35.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182495 2024-09-17T16:11:35.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509548 3-343597383716 4-107374182495' 2024-09-17T16:11:35.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:35.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:11:36.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=369367187487 2024-09-17T16:11:36.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 369367187487 2024-09-17T16:11:36.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509548 3-343597383716 4-107374182495 5-369367187487' 2024-09-17T16:11:36.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:36.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:11:36.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=442381631513 2024-09-17T16:11:36.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 442381631513 2024-09-17T16:11:36.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509548 3-343597383716 4-107374182495 5-369367187487 6-442381631513' 2024-09-17T16:11:36.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:36.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T16:11:36.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691912 2024-09-17T16:11:36.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691912 2024-09-17T16:11:36.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509548 3-343597383716 4-107374182495 5-369367187487 6-442381631513 7-171798691912' 2024-09-17T16:11:36.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:36.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T16:11:36.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528387 2024-09-17T16:11:36.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528387 2024-09-17T16:11:36.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509548 3-343597383716 4-107374182495 5-369367187487 6-442381631513 7-171798691912 8-193273528387' 2024-09-17T16:11:36.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:36.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T16:11:36.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=476741369876 2024-09-17T16:11:36.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 476741369876 2024-09-17T16:11:36.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509548 3-343597383716 4-107374182495 5-369367187487 6-442381631513 7-171798691912 8-193273528387 9-476741369876' 2024-09-17T16:11:36.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:36.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-17T16:11:37.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201337 2024-09-17T16:11:37.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201337 2024-09-17T16:11:37.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836605 1-42949673079 2-64424509548 3-343597383716 4-107374182495 5-369367187487 6-442381631513 7-171798691912 8-193273528387 9-476741369876 10-236223201337' 2024-09-17T16:11:37.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:37.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836605 2024-09-17T16:11:37.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:37.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:11:37.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836605 2024-09-17T16:11:37.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:37.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836605 2024-09-17T16:11:37.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836605' 2024-09-17T16:11:37.091 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836605 2024-09-17T16:11:37.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:11:37.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836605 -lt 21474836605 2024-09-17T16:11:37.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:37.571 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673079 2024-09-17T16:11:37.571 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:37.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:11:37.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673079 2024-09-17T16:11:37.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:37.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673079 2024-09-17T16:11:37.573 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949673079 2024-09-17T16:11:37.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673079' 2024-09-17T16:11:37.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:11:38.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673079 -lt 42949673079 2024-09-17T16:11:38.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:38.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509548 2024-09-17T16:11:38.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:38.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:11:38.110 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509548 2024-09-17T16:11:38.110 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:38.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509548 2024-09-17T16:11:38.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509548' 2024-09-17T16:11:38.111 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509548 2024-09-17T16:11:38.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:11:38.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509549 -lt 64424509548 2024-09-17T16:11:38.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:38.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-343597383716 2024-09-17T16:11:38.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:38.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:11:38.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-343597383716 2024-09-17T16:11:38.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:38.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383716 2024-09-17T16:11:38.538 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 343597383716 2024-09-17T16:11:38.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 343597383716' 2024-09-17T16:11:38.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:11:39.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383717 -lt 343597383716 2024-09-17T16:11:39.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:39.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182495 2024-09-17T16:11:39.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:39.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:11:39.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182495 2024-09-17T16:11:39.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:39.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182495 2024-09-17T16:11:39.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182495' 2024-09-17T16:11:39.015 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182495 2024-09-17T16:11:39.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:11:39.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182496 -lt 107374182495 2024-09-17T16:11:39.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:39.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-369367187487 2024-09-17T16:11:39.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:39.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:11:39.589 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-369367187487 2024-09-17T16:11:39.589 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:39.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=369367187487 2024-09-17T16:11:39.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 369367187487' 2024-09-17T16:11:39.590 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 369367187487 2024-09-17T16:11:39.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:11:39.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 369367187488 -lt 369367187487 2024-09-17T16:11:39.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:39.998 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-442381631513 2024-09-17T16:11:39.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:39.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:11:40.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-442381631513 2024-09-17T16:11:40.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:40.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=442381631513 2024-09-17T16:11:40.001 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 442381631513 2024-09-17T16:11:40.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 442381631513' 2024-09-17T16:11:40.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:11:40.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 442381631513 -lt 442381631513 2024-09-17T16:11:40.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:40.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691912 2024-09-17T16:11:40.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:40.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T16:11:40.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691912 2024-09-17T16:11:40.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:40.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691912 2024-09-17T16:11:40.410 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691912 2024-09-17T16:11:40.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691912' 2024-09-17T16:11:40.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T16:11:40.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691913 -lt 171798691912 2024-09-17T16:11:40.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:40.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528387 2024-09-17T16:11:40.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:40.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T16:11:40.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528387 2024-09-17T16:11:40.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:40.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528387 2024-09-17T16:11:40.802 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 193273528387 2024-09-17T16:11:40.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528387' 2024-09-17T16:11:40.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T16:11:41.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528388 -lt 193273528387 2024-09-17T16:11:41.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:41.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-476741369876 2024-09-17T16:11:41.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:41.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T16:11:41.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-476741369876 2024-09-17T16:11:41.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:41.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=476741369876 2024-09-17T16:11:41.232 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.9 seq 476741369876 2024-09-17T16:11:41.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 476741369876' 2024-09-17T16:11:41.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T16:11:41.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 476741369876 -lt 476741369876 2024-09-17T16:11:41.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:41.619 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201337 2024-09-17T16:11:41.619 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:41.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-17T16:11:41.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201337 2024-09-17T16:11:41.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:41.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201337 2024-09-17T16:11:41.621 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.10 seq 236223201337 2024-09-17T16:11:41.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201337' 2024-09-17T16:11:41.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-17T16:11:42.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201337 -lt 236223201337 2024-09-17T16:11:42.022 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:11:42.022 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:11:42.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:11:42.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 28 == 0 2024-09-17T16:11:42.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:11:42.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:11:42.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:11:42.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:11:42.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:11:42.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:11:42.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:11:42.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=28 2024-09-17T16:11:42.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:11:42.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:11:42.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:11:43.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 = 28 2024-09-17T16:11:43.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:11:43.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:11:43.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-09-17T16:11:43.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-17T16:11:43.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:11:43.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:11:43.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<5\>' 2024-09-17T16:11:43.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-17T16:11:43.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:11:43.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-17T16:11:43.955 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:11:43.955 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:11:43.956 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:11:43.956 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:11:43.956 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-17T16:11:43.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 7 9 4 3 10 2024-09-17T16:11:43.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-09-17T16:11:43.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-17T16:11:43.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:11:43.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:11:43.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<1\>' 2024-09-17T16:11:43.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-17T16:11:43.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:11:44.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-17T16:11:44.415 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:11:44.416 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:11:44.416 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:11:44.416 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:11:44.416 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-17T16:11:44.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 7 9 4 3 10 2024-09-17T16:11:44.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:140: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-17T16:11:44.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:141: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:11:44.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-09-17T16:11:44.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 5 2024-09-17T16:11:44.953 INFO:tasks.workunit.client.0.smithi162.stderr:osd.5 is already in. 2024-09-17T16:11:44.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-09-17T16:11:44.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 1 2024-09-17T16:11:46.781 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 is already in. 2024-09-17T16:11:46.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:150: rados_osds_out_in: wait_for_clean 2024-09-17T16:11:46.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:11:46.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:11:46.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:11:46.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:11:46.794 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:11:46.794 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:11:46.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:11:46.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:11:46.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:11:46.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:11:46.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:11:46.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:11:46.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:11:46.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:11:46.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:11:47.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:11:47.621 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:11:47.621 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:11:47.621 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:11:47.621 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:11:47.621 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:11:47.621 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:11:47.621 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:11:47.621 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-09-17T16:11:47.622 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:11:47.622 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-17T16:11:47.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:11:47.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:47.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:11:47.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836609 2024-09-17T16:11:47.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836609 2024-09-17T16:11:47.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836609' 2024-09-17T16:11:47.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:47.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:11:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673083 2024-09-17T16:11:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673083 2024-09-17T16:11:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836609 1-42949673083' 2024-09-17T16:11:48.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:48.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:11:48.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509553 2024-09-17T16:11:48.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509553 2024-09-17T16:11:48.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509553' 2024-09-17T16:11:48.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:48.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:11:48.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383721 2024-09-17T16:11:48.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383721 2024-09-17T16:11:48.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509553 3-343597383721' 2024-09-17T16:11:48.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:48.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:11:48.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182500 2024-09-17T16:11:48.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182500 2024-09-17T16:11:48.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509553 3-343597383721 4-107374182500' 2024-09-17T16:11:48.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:48.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:11:49.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=369367187492 2024-09-17T16:11:49.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 369367187492 2024-09-17T16:11:49.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509553 3-343597383721 4-107374182500 5-369367187492' 2024-09-17T16:11:49.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:49.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:11:49.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=442381631517 2024-09-17T16:11:49.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 442381631517 2024-09-17T16:11:49.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509553 3-343597383721 4-107374182500 5-369367187492 6-442381631517' 2024-09-17T16:11:49.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:49.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T16:11:49.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691917 2024-09-17T16:11:49.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691917 2024-09-17T16:11:49.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509553 3-343597383721 4-107374182500 5-369367187492 6-442381631517 7-171798691917' 2024-09-17T16:11:49.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:49.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T16:11:49.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528392 2024-09-17T16:11:49.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528392 2024-09-17T16:11:49.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509553 3-343597383721 4-107374182500 5-369367187492 6-442381631517 7-171798691917 8-193273528392' 2024-09-17T16:11:49.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:49.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T16:11:49.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=476741369880 2024-09-17T16:11:49.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 476741369880 2024-09-17T16:11:49.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509553 3-343597383721 4-107374182500 5-369367187492 6-442381631517 7-171798691917 8-193273528392 9-476741369880' 2024-09-17T16:11:49.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:11:49.487 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-17T16:11:49.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201341 2024-09-17T16:11:49.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201341 2024-09-17T16:11:49.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509553 3-343597383721 4-107374182500 5-369367187492 6-442381631517 7-171798691917 8-193273528392 9-476741369880 10-236223201341' 2024-09-17T16:11:49.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:49.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836609 2024-09-17T16:11:49.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:49.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:11:49.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836609 2024-09-17T16:11:49.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:49.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836609 2024-09-17T16:11:49.604 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836609 2024-09-17T16:11:49.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836609' 2024-09-17T16:11:49.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:11:50.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836609 -lt 21474836609 2024-09-17T16:11:50.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:50.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673083 2024-09-17T16:11:50.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:50.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:11:50.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673083 2024-09-17T16:11:50.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:50.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673083 2024-09-17T16:11:50.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673083' 2024-09-17T16:11:50.042 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949673083 2024-09-17T16:11:50.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:11:50.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673084 -lt 42949673083 2024-09-17T16:11:50.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:50.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509553 2024-09-17T16:11:50.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:50.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:11:50.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509553 2024-09-17T16:11:50.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:50.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509553 2024-09-17T16:11:50.432 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509553 2024-09-17T16:11:50.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509553' 2024-09-17T16:11:50.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:11:50.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509553 -lt 64424509553 2024-09-17T16:11:50.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:50.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-343597383721 2024-09-17T16:11:50.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:50.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:11:50.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-343597383721 2024-09-17T16:11:50.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:50.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383721 2024-09-17T16:11:50.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 343597383721' 2024-09-17T16:11:50.855 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 343597383721 2024-09-17T16:11:50.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:11:51.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383721 -lt 343597383721 2024-09-17T16:11:51.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:51.246 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182500 2024-09-17T16:11:51.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:51.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:11:51.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182500 2024-09-17T16:11:51.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:51.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182500 2024-09-17T16:11:51.249 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182500 2024-09-17T16:11:51.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182500' 2024-09-17T16:11:51.250 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:11:51.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182500 -lt 107374182500 2024-09-17T16:11:51.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:51.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-369367187492 2024-09-17T16:11:51.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:51.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:11:51.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-369367187492 2024-09-17T16:11:51.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:51.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=369367187492 2024-09-17T16:11:51.691 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 369367187492 2024-09-17T16:11:51.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 369367187492' 2024-09-17T16:11:51.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:11:52.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 369367187492 -lt 369367187492 2024-09-17T16:11:52.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:52.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-442381631517 2024-09-17T16:11:52.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:52.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:11:52.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-442381631517 2024-09-17T16:11:52.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:52.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=442381631517 2024-09-17T16:11:52.116 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 442381631517 2024-09-17T16:11:52.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 442381631517' 2024-09-17T16:11:52.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:11:52.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 442381631517 -lt 442381631517 2024-09-17T16:11:52.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:52.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691917 2024-09-17T16:11:52.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:52.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T16:11:52.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691917 2024-09-17T16:11:52.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:52.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691917 2024-09-17T16:11:52.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691917' 2024-09-17T16:11:52.545 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691917 2024-09-17T16:11:52.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T16:11:52.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691917 -lt 171798691917 2024-09-17T16:11:52.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:52.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528392 2024-09-17T16:11:52.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:52.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T16:11:52.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528392 2024-09-17T16:11:52.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:52.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528392 2024-09-17T16:11:52.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528392' 2024-09-17T16:11:52.930 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 193273528392 2024-09-17T16:11:52.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T16:11:53.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528393 -lt 193273528392 2024-09-17T16:11:53.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:53.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-476741369880 2024-09-17T16:11:53.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:53.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T16:11:53.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-476741369880 2024-09-17T16:11:53.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:53.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=476741369880 2024-09-17T16:11:53.348 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.9 seq 476741369880 2024-09-17T16:11:53.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 476741369880' 2024-09-17T16:11:53.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T16:11:53.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 476741369881 -lt 476741369880 2024-09-17T16:11:53.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:11:53.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201341 2024-09-17T16:11:53.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:11:53.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-17T16:11:53.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201341 2024-09-17T16:11:53.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:11:53.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201341 2024-09-17T16:11:53.775 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.10 seq 236223201341 2024-09-17T16:11:53.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201341' 2024-09-17T16:11:53.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-17T16:11:54.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201342 -lt 236223201341 2024-09-17T16:11:54.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:11:54.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:11:54.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:11:54.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 28 == 0 2024-09-17T16:11:54.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:11:54.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:11:54.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:11:54.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:11:54.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:11:54.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:11:54.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:11:55.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=28 2024-09-17T16:11:55.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:11:55.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:11:55.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:11:55.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 = 28 2024-09-17T16:11:55.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:11:55.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:11:55.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-17T16:11:55.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:11:55.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:11:55.660 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-17T16:11:55.660 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:11:56.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-17T16:11:56.098 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:11:56.098 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:11:56.098 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:11:56.098 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:11:56.098 INFO:tasks.workunit.client.0.smithi162.stderr:10' 2024-09-17T16:11:56.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 10 2024-09-17T16:11:56.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: test '5 1 9 4 3 10' = '5 1 9 4 3 10' 2024-09-17T16:11:56.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:152: rados_osds_out_in: rm td/test-erasure-code/ORIGINAL 2024-09-17T16:11:56.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:229: TEST_rados_put_get_jerasure: delete_pool pool-jerasure 2024-09-17T16:11:56.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-jerasure 2024-09-17T16:11:56.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-17T16:11:56.578 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-17T16:11:56.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:230: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile rm profile-jerasure 2024-09-17T16:11:58.033 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile profile-jerasure does not exist 2024-09-17T16:11:58.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-17T16:11:58.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_advanced td/test-erasure-code 2024-09-17T16:11:58.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:156: TEST_rados_put_get_lrc_advanced: local dir=td/test-erasure-code 2024-09-17T16:11:58.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:157: TEST_rados_put_get_lrc_advanced: local poolname=pool-lrc-a 2024-09-17T16:11:58.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:158: TEST_rados_put_get_lrc_advanced: local profile=profile-lrc-a 2024-09-17T16:11:58.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:160: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile set profile-lrc-a plugin=lrc mapping=DD_ 'crush-steps=[ [ "chooseleaf", "osd", 0 ] ]' 'layers=[ [ "DDc", "" ] ]' 2024-09-17T16:11:58.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:165: TEST_rados_put_get_lrc_advanced: create_pool pool-lrc-a 12 12 erasure profile-lrc-a 2024-09-17T16:11:58.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-lrc-a 12 12 erasure profile-lrc-a 2024-09-17T16:11:59.297 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-lrc-a' already exists 2024-09-17T16:11:59.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:12:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:168: TEST_rados_put_get_lrc_advanced: rados_put_get td/test-erasure-code pool-lrc-a 2024-09-17T16:12:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-17T16:12:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc-a 2024-09-17T16:12:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-17T16:12:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:12:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-17T16:12:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:12:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-17T16:12:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:12:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-17T16:12:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:12:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-17T16:12:00.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc-a put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-17T16:12:02.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-09-17T16:12:02.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:12:02.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-17T16:12:02.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-09-17T16:12:02.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc-a 2024-09-17T16:12:02.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:12:02.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-09-17T16:12:02.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:12:03.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-17T16:12:03.253 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:12:03.253 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:12:03.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 9 3 2024-09-17T16:12:03.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '9' '3') 2024-09-17T16:12:03.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-17T16:12:03.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-09-17T16:12:03.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 3 2024-09-17T16:12:03.778 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 is already out. 2024-09-17T16:12:03.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-17T16:12:08.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-09-17T16:12:08.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc-a 2024-09-17T16:12:08.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:12:08.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<3\>' 2024-09-17T16:12:08.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-09-17T16:12:08.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:12:09.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-17T16:12:09.227 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:12:09.227 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2024-09-17T16:12:09.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 9 7 2024-09-17T16:12:09.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-09-17T16:12:09.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:12:09.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 3 2024-09-17T16:12:09.764 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 is already in. 2024-09-17T16:12:09.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-17T16:12:09.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:170: TEST_rados_put_get_lrc_advanced: delete_pool pool-lrc-a 2024-09-17T16:12:09.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-lrc-a 2024-09-17T16:12:09.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-lrc-a pool-lrc-a --yes-i-really-really-mean-it 2024-09-17T16:12:11.564 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-lrc-a' does not exist 2024-09-17T16:12:11.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:171: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile rm profile-lrc-a 2024-09-17T16:12:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile profile-lrc-a does not exist 2024-09-17T16:12:12.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-17T16:12:12.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_kml td/test-erasure-code 2024-09-17T16:12:12.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:175: TEST_rados_put_get_lrc_kml: local dir=td/test-erasure-code 2024-09-17T16:12:12.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:176: TEST_rados_put_get_lrc_kml: local poolname=pool-lrc 2024-09-17T16:12:12.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:177: TEST_rados_put_get_lrc_kml: local profile=profile-lrc 2024-09-17T16:12:12.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:179: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile set profile-lrc plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd 2024-09-17T16:12:13.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:183: TEST_rados_put_get_lrc_kml: create_pool pool-lrc 12 12 erasure profile-lrc 2024-09-17T16:12:13.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-lrc 12 12 erasure profile-lrc 2024-09-17T16:12:14.522 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-lrc' already exists 2024-09-17T16:12:14.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:12:15.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:186: TEST_rados_put_get_lrc_kml: rados_put_get td/test-erasure-code pool-lrc 2024-09-17T16:12:15.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-17T16:12:15.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc 2024-09-17T16:12:15.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-17T16:12:15.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:12:15.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-17T16:12:15.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:12:15.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-17T16:12:15.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:12:15.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-17T16:12:15.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:12:15.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-17T16:12:15.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-17T16:12:17.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-09-17T16:12:17.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:12:17.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-17T16:12:17.873 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc SOMETHING 2024-09-17T16:12:17.873 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc 2024-09-17T16:12:17.873 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:12:17.874 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-09-17T16:12:17.874 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:12:18.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=8 2024-09-17T16:12:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:12:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:12:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:10 2024-09-17T16:12:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:12:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:12:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:12:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2024-09-17T16:12:18.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 8 6 0 10 5 3 9 7 2024-09-17T16:12:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('8' '6' '0' '10' '5' '3' '9' '7') 2024-09-17T16:12:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-17T16:12:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=7 2024-09-17T16:12:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2024-09-17T16:12:19.835 INFO:tasks.workunit.client.0.smithi162.stderr:osd.7 is already out. 2024-09-17T16:12:19.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-17T16:12:24.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc SOMETHING 2024-09-17T16:12:24.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc 2024-09-17T16:12:24.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:12:24.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<7\>' 2024-09-17T16:12:24.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-09-17T16:12:24.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:12:25.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=8 2024-09-17T16:12:25.297 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:12:25.297 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:12:25.297 INFO:tasks.workunit.client.0.smithi162.stderr:10 2024-09-17T16:12:25.297 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:12:25.298 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:12:25.298 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:12:25.298 INFO:tasks.workunit.client.0.smithi162.stderr:4' 2024-09-17T16:12:25.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 8 6 0 10 5 3 9 4 2024-09-17T16:12:25.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-09-17T16:12:25.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:12:25.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 7 2024-09-17T16:12:25.853 INFO:tasks.workunit.client.0.smithi162.stderr:osd.7 is already in. 2024-09-17T16:12:25.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-17T16:12:25.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:188: TEST_rados_put_get_lrc_kml: delete_pool pool-lrc 2024-09-17T16:12:25.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-lrc 2024-09-17T16:12:25.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-lrc pool-lrc --yes-i-really-really-mean-it 2024-09-17T16:12:27.507 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-lrc' does not exist 2024-09-17T16:12:27.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:189: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile rm profile-lrc 2024-09-17T16:12:29.138 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile profile-lrc does not exist 2024-09-17T16:12:29.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-17T16:12:29.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_shec td/test-erasure-code 2024-09-17T16:12:29.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:234: TEST_rados_put_get_shec: local dir=td/test-erasure-code 2024-09-17T16:12:29.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:236: TEST_rados_put_get_shec: local poolname=pool-shec 2024-09-17T16:12:29.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:237: TEST_rados_put_get_shec: local profile=profile-shec 2024-09-17T16:12:29.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:239: TEST_rados_put_get_shec: ceph osd erasure-code-profile set profile-shec plugin=shec k=2 m=1 c=1 crush-failure-domain=osd 2024-09-17T16:12:29.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:243: TEST_rados_put_get_shec: create_pool pool-shec 12 12 erasure profile-shec 2024-09-17T16:12:29.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-shec 12 12 erasure profile-shec 2024-09-17T16:12:30.641 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-shec' already exists 2024-09-17T16:12:30.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:12:31.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:246: TEST_rados_put_get_shec: rados_put_get td/test-erasure-code pool-shec 2024-09-17T16:12:31.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-17T16:12:31.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-shec 2024-09-17T16:12:31.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-17T16:12:31.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:12:31.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-17T16:12:31.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:12:31.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-17T16:12:31.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:12:31.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-17T16:12:31.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-17T16:12:31.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-17T16:12:31.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-shec put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-17T16:12:33.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-09-17T16:12:33.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:12:33.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-17T16:12:33.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-shec SOMETHING 2024-09-17T16:12:33.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-shec 2024-09-17T16:12:33.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:12:33.793 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-09-17T16:12:33.793 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:12:34.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=7 2024-09-17T16:12:34.309 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:12:34.309 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-09-17T16:12:34.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 7 9 1 2024-09-17T16:12:34.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('7' '9' '1') 2024-09-17T16:12:34.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-17T16:12:34.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-09-17T16:12:34.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 1 2024-09-17T16:12:35.127 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 is already out. 2024-09-17T16:12:35.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-17T16:12:40.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-shec SOMETHING 2024-09-17T16:12:40.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-shec 2024-09-17T16:12:40.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T16:12:40.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<1\>' 2024-09-17T16:12:40.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-09-17T16:12:40.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:12:40.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=7 2024-09-17T16:12:40.580 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-09-17T16:12:40.580 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-17T16:12:40.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 7 9 6 2024-09-17T16:12:40.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-09-17T16:12:40.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-17T16:12:40.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 1 2024-09-17T16:12:41.145 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 is already in. 2024-09-17T16:12:41.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-17T16:12:41.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:248: TEST_rados_put_get_shec: delete_pool pool-shec 2024-09-17T16:12:41.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-shec 2024-09-17T16:12:41.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-shec pool-shec --yes-i-really-really-mean-it 2024-09-17T16:12:42.379 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-shec' does not exist 2024-09-17T16:12:42.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:249: TEST_rados_put_get_shec: ceph osd erasure-code-profile rm profile-shec 2024-09-17T16:12:43.749 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile profile-shec does not exist 2024-09-17T16:12:43.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:51: run: delete_pool ecpool 2024-09-17T16:12:43.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=ecpool 2024-09-17T16:12:43.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete ecpool ecpool --yes-i-really-really-mean-it 2024-09-17T16:12:44.465 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' does not exist 2024-09-17T16:12:44.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:52: run: teardown td/test-erasure-code 2024-09-17T16:12:44.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code 2024-09-17T16:12:44.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:12:44.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code KILL 2024-09-17T16:12:44.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:12:44.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:12:44.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:12:44.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:12:44.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:12:44.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:12:44.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:12:44.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:12:44.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:12:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:12:44.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:12:44.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:12:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:12:44.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:12:44.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:12:44.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:12:44.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:12:44.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:12:44.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code 2024-09-17T16:12:44.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:12:44.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:12:44.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:12:44.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56062 2024-09-17T16:12:44.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:12:44.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:12:44.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-17T16:12:44.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/test-erasure-code 0 2024-09-17T16:12:44.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code 2024-09-17T16:12:44.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-17T16:12:44.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code KILL 2024-09-17T16:12:44.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:12:44.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:12:44.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:12:44.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:12:44.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:12:44.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:12:44.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:12:44.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:12:44.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:12:44.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:12:44.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:12:44.702 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:12:44.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:12:44.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:12:44.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:12:44.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:12:44.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:12:44.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-17T16:12:44.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code 2024-09-17T16:12:44.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:12:44.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:12:44.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56062 2024-09-17T16:12:44.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56062 2024-09-17T16:12:44.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:12:44.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:12:44.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-17T16:12:44.708 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T16:12:44.709 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T16:12:44.734 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-eio.sh... 2024-09-17T16:12:44.734 DEBUG:teuthology.orchestra.run.smithi162:workunit test erasure-code/test-erasure-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh 2024-09-17T16:12:44.795 INFO:tasks.workunit.client.0.smithi162.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-17T16:12:44.801 INFO:tasks.workunit.client.0.smithi162.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-17T16:12:44.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T16:12:44.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T16:12:44.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-17T16:12:44.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-17T16:12:44.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-17T16:12:44.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-17T16:12:44.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-17T16:12:44.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-17T16:12:44.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/test-erasure-eio 2024-09-17T16:12:44.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:22: run: local dir=td/test-erasure-eio 2024-09-17T16:12:44.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:23: run: shift 2024-09-17T16:12:44.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: export CEPH_MON=127.0.0.1:7112 2024-09-17T16:12:44.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: CEPH_MON=127.0.0.1:7112 2024-09-17T16:12:44.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:26: run: export CEPH_ARGS 2024-09-17T16:12:44.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: uuidgen 2024-09-17T16:12:44.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: CEPH_ARGS+='--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none ' 2024-09-17T16:12:44.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7112 ' 2024-09-17T16:12:44.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-09-17T16:12:44.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: set 2024-09-17T16:12:44.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-17T16:12:44.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: local 'funcs=TEST_ec_backfill_unfound 2024-09-17T16:12:44.805 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_ec_object_attr_read_error 2024-09-17T16:12:44.805 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_ec_recovery_multiple_errors 2024-09-17T16:12:44.805 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_ec_recovery_multiple_objects 2024-09-17T16:12:44.805 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_ec_recovery_multiple_objects_eio 2024-09-17T16:12:44.805 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_ec_recovery_unfound 2024-09-17T16:12:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_ec_single_recovery_error 2024-09-17T16:12:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_get_bad_size_shard_0 2024-09-17T16:12:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_get_bad_size_shard_1 2024-09-17T16:12:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_get_subread_eio_shard_0 2024-09-17T16:12:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_get_subread_eio_shard_1 2024-09-17T16:12:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_get_subread_missing 2024-09-17T16:12:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_get_with_subreadall_eio_shard_0 2024-09-17T16:12:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_rados_get_with_subreadall_eio_shard_1' 2024-09-17T16:12:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-17T16:12:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-17T16:12:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-17T16:12:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-17T16:12:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:12:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:12:44.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:12:44.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:12:44.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:12:44.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:12:44.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:12:44.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:12:44.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:12:44.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:12:44.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:12:44.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:12:44.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:12:44.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:12:44.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:12:44.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:12:44.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:12:44.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:12:44.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:12:44.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:12:44.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:12:44.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:12:44.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:12:44.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:12:44.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:12:44.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:12:44.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:12:44.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:12:44.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-17T16:12:44.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:12:44.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:12:44.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:12:44.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.87940 2024-09-17T16:12:44.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:12:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-17T16:12:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-17T16:12:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:12:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-17T16:12:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-17T16:12:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-17T16:12:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:12:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:12:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:12:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-17T16:12:44.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-17T16:12:45.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:12:45.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:12:45.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:12:45.280 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:12:45.280 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:12:45.280 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:12:45.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:12:45.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T16:12:45.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:12:45.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:12:45.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:12:45.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:12:45.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:12:45.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:12:45.458 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:12:45.458 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:12:45.458 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:12:45.458 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:12:45.458 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:12:45.458 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:12:45.459 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:12:45.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:12:45.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get fsid 2024-09-17T16:12:45.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:12:45.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:12:45.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:12:45.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:12:45.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:12:45.539 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:12:45.539 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:12:45.539 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:12:45.539 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:12:45.539 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:12:45.540 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:12:45.540 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:12:45.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:12:45.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get mon_host 2024-09-17T16:12:45.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-17T16:12:45.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-17T16:12:45.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:12:45.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:12:45.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:12:45.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-17T16:12:45.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:12:46.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:12:46.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:12:46.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:12:46.470 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:12:46.470 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:12:46.470 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:12:46.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:12:46.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:12:46.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:12:46.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-17T16:12:46.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T16:12:46.814 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T16:12:46.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:12:47.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-17T16:12:47.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:12:47.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:12:47.827 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:12:47.827 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:12:47.827 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:12:47.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:12:47.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-17T16:12:47.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-mon.a.asok log flush 2024-09-17T16:12:47.898 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:12:47.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-17T16:12:47.907 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:12:45.312+0000 7f9532b22c80 0 load: jerasure load: lrc 2024-09-17T16:12:47.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_backfill_unfound td/test-erasure-eio 2024-09-17T16:12:47.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:531: TEST_ec_backfill_unfound: local dir=td/test-erasure-eio 2024-09-17T16:12:47.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:532: TEST_ec_backfill_unfound: local objname=myobject 2024-09-17T16:12:47.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:533: TEST_ec_backfill_unfound: local lastobj=300 2024-09-17T16:12:47.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:535: TEST_ec_backfill_unfound: local testobj=obj250 2024-09-17T16:12:47.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: ORIG_ARGS='--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:12:47.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:538: TEST_ec_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-17T16:12:47.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: setup_osds 5 2024-09-17T16:12:47.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-09-17T16:12:47.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-17T16:12:47.909 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-09-17T16:12:47.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-09-17T16:12:47.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:12:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-17T16:12:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:12:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:12:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:12:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:12:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-17T16:12:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-17T16:12:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:12:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:12:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:12:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-17T16:12:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:12:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:12:47.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:12:47.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:12:47.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:12:47.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:12:47.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:12:47.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-17T16:12:47.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:12:47.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=86d9ade7-2192-49f5-9764-758d2da6fca5 2024-09-17T16:12:47.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 86d9ade7-2192-49f5-9764-758d2da6fca5' 2024-09-17T16:12:47.916 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 86d9ade7-2192-49f5-9764-758d2da6fca5 2024-09-17T16:12:47.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:12:47.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB/qulmWuyqNxAAqgCOHSNat6crBZ2FpFbZHA== 2024-09-17T16:12:47.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB/qulmWuyqNxAAqgCOHSNat6crBZ2FpFbZHA=="}' 2024-09-17T16:12:47.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 86d9ade7-2192-49f5-9764-758d2da6fca5 -i td/test-erasure-eio/0/new.json 2024-09-17T16:12:48.284 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:12:48.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-17T16:12:48.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/qulmWuyqNxAAqgCOHSNat6crBZ2FpFbZHA== --osd-uuid 86d9ade7-2192-49f5-9764-758d2da6fca5 2024-09-17T16:12:48.327 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:12:48.325+0000 7f1ee8c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:12:48.351 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:12:48.349+0000 7f1ee8c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:12:48.352 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:12:48.350+0000 7f1ee8c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:12:48.352 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:12:48.350+0000 7f1ee8c16640 -1 bdev(0x562356ecfc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:12:48.352 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:12:48.351+0000 7f1ee8c16640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-17T16:12:53.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-17T16:12:53.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:12:53.999 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:12:53.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:12:53.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:12:54.541 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:12:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:12:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:12:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:12:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:12:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:12:54.572 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:12:54.570+0000 7fc9697a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:12:54.582 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:12:54.580+0000 7fc9697a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:12:54.583 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:12:54.582+0000 7fc9697a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:12:54.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:12:54.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:12:54.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:12:54.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:12:54.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:12:54.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:12:54.989 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:12:54.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:12:54.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:12:54.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:12:55.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:12:55.676 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:12:55.674+0000 7fc9697a6640 -1 Falling back to public interface 2024-09-17T16:12:56.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:12:56.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:12:56.415 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:12:56.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:12:56.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:12:56.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:12:56.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:12:57.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:12:57.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:12:57.849 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:12:57.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:12:57.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:12:57.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:12:58.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:12:58.718 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:12:58.716+0000 7fc9697a6640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:12:59.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:12:59.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:12:59.245 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:12:59.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:12:59.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:12:59.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:12:59.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:00.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:00.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:00.740 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:13:00.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:13:00.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:00.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:13:01.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:02.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:02.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:02.146 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:13:02.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:13:02.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:02.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:13:02.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:03.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:03.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:03.594 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:13:03.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:13:03.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:03.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:13:04.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:05.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:05.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:05.028 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:13:05.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:13:05.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:05.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:13:05.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:06.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:06.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:06.443 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:13:06.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:13:06.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:06.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:13:06.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:07.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:07.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:07.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:13:07.855 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:13:07.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:07.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:13:08.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:09.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:09.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:09.254 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:13:09.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:13:09.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:09.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:13:09.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:10.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:10.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:10.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:13:10.716 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:13:10.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:10.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:13:11.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:12.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:12.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:12.412 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:13:12.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:13:12.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:12.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:13:12.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:13.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:13.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:13.923 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:13:13.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:13:13.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:13.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:13:14.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:15.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:15.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:15.683 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:13:15.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:13:15.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:15.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:13:16.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:17.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:17.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:17.484 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:13:17.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:13:17.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:17.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:13:18.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:19.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:19.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:19.255 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:13:19.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:13:19.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:19.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:13:20.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:21.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:21.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:21.304 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:13:21.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:13:21.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:21.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:13:21.716 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/960297747,v1:127.0.0.1:6803/960297747] [v2:127.0.0.1:6804/960297747,v1:127.0.0.1:6805/960297747] exists,up 86d9ade7-2192-49f5-9764-758d2da6fca5 2024-09-17T16:13:21.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:13:21.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:13:21.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:13:21.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:13:21.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-17T16:13:21.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:13:21.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:13:21.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T16:13:21.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:13:21.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:13:21.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-17T16:13:21.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:13:21.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:13:21.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:13:21.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:13:21.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:13:21.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:13:21.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:13:21.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:13:21.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:13:21.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:13:21.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:13:21.722 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:13:21.722 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:13:21.722 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:13:21.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:13:21.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:13:21.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:13:21.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:13:21.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:13:21.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:13:21.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:13:21.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:13:21.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:13:21.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:13:21.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:13:21.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:13:21.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:13:21.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:13:21.723 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:13:21.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6066e0bb-bd40-43ca-9f5e-520569b50252 2024-09-17T16:13:21.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 6066e0bb-bd40-43ca-9f5e-520569b50252' 2024-09-17T16:13:21.724 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 6066e0bb-bd40-43ca-9f5e-520569b50252 2024-09-17T16:13:21.724 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:13:21.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQChqulmmekULBAA1Mg8g+cGFQkZLOUMQqAMgA== 2024-09-17T16:13:21.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQChqulmmekULBAA1Mg8g+cGFQkZLOUMQqAMgA=="}' 2024-09-17T16:13:21.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6066e0bb-bd40-43ca-9f5e-520569b50252 -i td/test-erasure-eio/1/new.json 2024-09-17T16:13:22.301 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:13:22.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-17T16:13:22.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQChqulmmekULBAA1Mg8g+cGFQkZLOUMQqAMgA== --osd-uuid 6066e0bb-bd40-43ca-9f5e-520569b50252 2024-09-17T16:13:22.345 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:13:22.343+0000 7fcd66a32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:13:22.346 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:13:22.345+0000 7fcd66a32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:13:22.348 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:13:22.346+0000 7fcd66a32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:13:22.348 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:13:22.346+0000 7fcd66a32640 -1 bdev(0x5582af395c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-17T16:13:22.348 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:13:22.346+0000 7fcd66a32640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-17T16:13:28.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-17T16:13:28.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:13:28.262 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T16:13:28.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T16:13:28.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:13:28.824 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T16:13:28.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T16:13:28.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:13:28.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:13:28.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:13:28.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:13:28.856 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:13:28.853+0000 7fc62b611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:13:28.865 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:13:28.862+0000 7fc62b611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:13:28.867 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:13:28.864+0000 7fc62b611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:13:29.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T16:13:29.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:13:29.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:13:29.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:13:29.261 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:13:29.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:13:29.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:29.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:13:29.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:29.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:29.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:30.219 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:13:30.216+0000 7fc62b611640 -1 Falling back to public interface 2024-09-17T16:13:30.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:30.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:30.697 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:13:30.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:13:30.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:30.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:31.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:32.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:32.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:32.114 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:13:32.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:13:32.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:32.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:32.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:13:33.156+0000 7fc62b611640 -1 osd.1 0 log_to_monitors true 2024-09-17T16:13:33.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:33.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:33.540 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:13:33.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:13:33.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:33.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:33.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:34.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:34.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:34.940 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:13:34.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:13:34.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:34.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:35.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:36.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:36.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:36.341 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:13:36.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:13:36.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:36.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:36.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:37.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:37.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:37.800 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:13:37.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:13:37.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:37.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:38.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:39.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:39.314 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:13:39.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:39.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:13:39.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:39.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:39.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:40.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:40.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:40.720 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:13:40.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:13:40.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:40.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:41.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:42.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:42.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:42.126 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:13:42.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:13:42.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:42.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:42.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:43.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:43.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:43.567 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:13:43.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:13:43.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:43.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:44.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:45.028 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:13:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:13:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:45.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:46.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:46.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:46.888 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:13:46.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:13:46.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:46.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:47.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:48.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:48.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:48.487 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:13:48.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:13:48.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:48.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:49.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:49.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:49.907 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:13:49.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:13:49.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:49.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:50.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:51.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:51.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:51.661 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:13:51.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:13:51.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:51.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:52.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:53.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:53.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:53.716 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:13:53.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:13:53.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:53.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:54.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:55.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:55.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:55.510 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:13:55.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:13:55.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:55.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:55.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:13:56.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:13:56.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:13:56.944 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:13:56.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:13:56.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:13:56.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:13:57.428 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2723903835,v1:127.0.0.1:6811/2723903835] [v2:127.0.0.1:6812/2723903835,v1:127.0.0.1:6813/2723903835] exists,up 6066e0bb-bd40-43ca-9f5e-520569b50252 2024-09-17T16:13:57.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:13:57.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:13:57.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:13:57.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:13:57.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-17T16:13:57.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:13:57.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:13:57.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T16:13:57.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:13:57.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:13:57.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-17T16:13:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:13:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:13:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:13:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:13:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:13:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:13:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:13:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:13:57.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:13:57.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:13:57.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:13:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:13:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:13:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:13:57.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:13:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:13:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:13:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:13:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:13:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:13:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:13:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:13:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:13:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:13:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:13:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:13:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:13:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:13:57.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:13:57.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d09e1f7c-4713-45c7-9eb5-9f689f2aca77 2024-09-17T16:13:57.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 d09e1f7c-4713-45c7-9eb5-9f689f2aca77' 2024-09-17T16:13:57.436 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 d09e1f7c-4713-45c7-9eb5-9f689f2aca77 2024-09-17T16:13:57.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:13:57.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDFqulmZ9DvGhAA8DueKFKk/WhZEw9FPEzOnA== 2024-09-17T16:13:57.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDFqulmZ9DvGhAA8DueKFKk/WhZEw9FPEzOnA=="}' 2024-09-17T16:13:57.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d09e1f7c-4713-45c7-9eb5-9f689f2aca77 -i td/test-erasure-eio/2/new.json 2024-09-17T16:13:57.939 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:13:57.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-17T16:13:57.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDFqulmZ9DvGhAA8DueKFKk/WhZEw9FPEzOnA== --osd-uuid d09e1f7c-4713-45c7-9eb5-9f689f2aca77 2024-09-17T16:13:57.984 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:13:57.982+0000 7fcb44c87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:13:57.986 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:13:57.984+0000 7fcb44c87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:13:57.987 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:13:57.985+0000 7fcb44c87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:13:57.987 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:13:57.985+0000 7fcb44c87640 -1 bdev(0x5640bac83c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-17T16:13:57.987 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:13:57.986+0000 7fcb44c87640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-17T16:14:03.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-17T16:14:03.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:14:03.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T16:14:03.870 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T16:14:03.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:14:04.413 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T16:14:04.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T16:14:04.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:14:04.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:14:04.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:14:04.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:14:04.444 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:14:04.443+0000 7f0288a0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:14:04.452 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:14:04.450+0000 7f0288a0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:14:04.453 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:14:04.452+0000 7f0288a0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:14:04.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T16:14:04.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:14:04.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:14:04.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:14:04.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:14:04.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:04.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:14:04.825 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:14:04.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:04.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:05.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:05.546 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:14:05.545+0000 7f0288a0b640 -1 Falling back to public interface 2024-09-17T16:14:06.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:06.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:06.216 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:14:06.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:14:06.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:06.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:06.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:07.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:07.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:07.670 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:14:07.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:14:07.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:07.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:08.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:08.272 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:14:08.271+0000 7f0288a0b640 -1 osd.2 0 log_to_monitors true 2024-09-17T16:14:09.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:09.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:09.073 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:14:09.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:14:09.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:09.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:09.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:10.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:10.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:10.476 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:14:10.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:14:10.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:10.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:11.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:12.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:12.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:12.005 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:14:12.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:14:12.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:12.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:12.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:13.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:13.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:13.476 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:14:13.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:14:13.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:13.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:13.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:14.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:14.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:14.876 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:14:14.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:14:14.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:14.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:15.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:16.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:16.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:16.457 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:14:16.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:14:16.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:16.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:16.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:17.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:17.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:17.933 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:14:17.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:14:17.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:17.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:18.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:19.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:19.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:19.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:14:19.361 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:14:19.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:19.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:19.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:20.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:20.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:20.776 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:14:20.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:14:20.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:20.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:21.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:22.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:22.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:22.188 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:14:22.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:14:22.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:22.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:22.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:23.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:23.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:23.708 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:14:23.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:14:23.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:23.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:24.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:25.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:25.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:25.435 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:14:25.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:14:25.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:25.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:26.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:27.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:27.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:27.305 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:14:27.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:14:27.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:27.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:28.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:29.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:29.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:29.015 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:14:29.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:14:29.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:29.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:30.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:31.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:31.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:31.046 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:14:31.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:14:31.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:31.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:14:31.473 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1989273976,v1:127.0.0.1:6819/1989273976] [v2:127.0.0.1:6820/1989273976,v1:127.0.0.1:6821/1989273976] exists,up d09e1f7c-4713-45c7-9eb5-9f689f2aca77 2024-09-17T16:14:31.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:14:31.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:14:31.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:14:31.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:14:31.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-17T16:14:31.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:14:31.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:14:31.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T16:14:31.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:14:31.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T16:14:31.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-17T16:14:31.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:14:31.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:14:31.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:14:31.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T16:14:31.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T16:14:31.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:14:31.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:14:31.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:14:31.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:14:31.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:14:31.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:14:31.479 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:14:31.479 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:14:31.479 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:14:31.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:14:31.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:14:31.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:14:31.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:14:31.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:14:31.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:14:31.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:14:31.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:14:31.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:14:31.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:14:31.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:14:31.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:14:31.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:14:31.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T16:14:31.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:14:31.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5bc4db15-26bc-46fd-a9d0-3339a56c78b7 2024-09-17T16:14:31.480 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 5bc4db15-26bc-46fd-a9d0-3339a56c78b7 2024-09-17T16:14:31.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 5bc4db15-26bc-46fd-a9d0-3339a56c78b7' 2024-09-17T16:14:31.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:14:31.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDnqulm8pSeHRAA4P6onPN7d3zmkGgJji0Wmw== 2024-09-17T16:14:31.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDnqulm8pSeHRAA4P6onPN7d3zmkGgJji0Wmw=="}' 2024-09-17T16:14:31.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5bc4db15-26bc-46fd-a9d0-3339a56c78b7 -i td/test-erasure-eio/3/new.json 2024-09-17T16:14:31.980 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:14:31.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-17T16:14:31.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnqulm8pSeHRAA4P6onPN7d3zmkGgJji0Wmw== --osd-uuid 5bc4db15-26bc-46fd-a9d0-3339a56c78b7 2024-09-17T16:14:32.025 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:14:32.023+0000 7f071e42c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:14:32.122 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:14:32.120+0000 7f071e42c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:14:32.123 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:14:32.121+0000 7f071e42c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:14:32.123 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:14:32.122+0000 7f071e42c640 -1 bdev(0x563fcb6dfc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-17T16:14:32.124 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:14:32.122+0000 7f071e42c640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-17T16:14:38.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-17T16:14:38.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:14:38.631 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-17T16:14:38.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T16:14:38.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:14:39.174 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T16:14:39.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T16:14:39.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:14:39.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:14:39.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:14:39.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:14:39.205 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:14:39.203+0000 7f6650285640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:14:39.210 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:14:39.208+0000 7f6650285640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:14:39.211 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:14:39.209+0000 7f6650285640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:14:39.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T16:14:39.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:14:39.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:14:39.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:14:39.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:14:39.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:39.624 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:14:39.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:14:39.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:39.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:14:40.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:40.302 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:14:40.301+0000 7f6650285640 -1 Falling back to public interface 2024-09-17T16:14:41.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:41.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:41.051 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:14:41.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:14:41.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:41.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:14:41.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:42.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:42.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:42.486 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:14:42.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:14:42.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:42.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:14:42.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:43.305 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:14:43.303+0000 7f6650285640 -1 osd.3 0 log_to_monitors true 2024-09-17T16:14:43.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:43.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:43.900 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:14:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:14:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:14:44.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:45.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:45.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:45.546 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:14:45.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:14:45.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:45.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:14:46.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:47.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:47.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:47.203 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:14:47.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:14:47.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:47.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:14:47.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:48.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:48.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:48.620 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:14:48.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:14:48.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:48.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:14:49.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:50.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:50.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:50.161 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:14:50.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:14:50.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:50.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:14:50.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:51.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:51.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:51.577 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:14:51.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:14:51.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:51.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:14:51.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:52.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:52.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:52.979 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:14:52.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:14:52.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:52.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:14:53.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:54.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:54.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:54.400 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:14:54.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:14:54.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:54.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:14:54.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:55.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:55.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:55.837 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:14:55.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:14:55.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:55.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:14:56.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:57.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:57.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:57.573 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:14:57.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:14:57.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:57.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:14:58.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:14:59.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:14:59.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:14:59.451 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:14:59.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:14:59.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:14:59.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:15:00.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:01.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:01.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:01.104 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:15:01.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:15:01.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:01.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:15:02.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:03.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:03.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:03.074 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:15:03.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:15:03.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:03.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:15:04.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:05.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:05.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:05.204 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:15:05.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:15:05.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:05.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:15:06.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:07.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:07.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:07.101 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:15:07.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:15:07.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:07.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:15:07.595 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/854248121,v1:127.0.0.1:6827/854248121] [v2:127.0.0.1:6828/854248121,v1:127.0.0.1:6829/854248121] exists,up 5bc4db15-26bc-46fd-a9d0-3339a56c78b7 2024-09-17T16:15:07.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:15:07.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:15:07.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:15:07.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:15:07.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-17T16:15:07.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:15:07.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:15:07.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-17T16:15:07.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:15:07.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-17T16:15:07.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-17T16:15:07.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:15:07.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:15:07.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:15:07.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-17T16:15:07.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-17T16:15:07.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:15:07.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:15:07.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:15:07.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:15:07.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:15:07.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:15:07.600 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:15:07.600 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:15:07.600 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:15:07.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:15:07.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:15:07.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:15:07.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:15:07.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:15:07.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:15:07.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:15:07.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:15:07.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:15:07.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:15:07.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:15:07.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:15:07.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:15:07.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-17T16:15:07.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:15:07.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1c0969d6-76b0-4bed-8825-36ef7650a841 2024-09-17T16:15:07.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 1c0969d6-76b0-4bed-8825-36ef7650a841' 2024-09-17T16:15:07.602 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 1c0969d6-76b0-4bed-8825-36ef7650a841 2024-09-17T16:15:07.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:15:07.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQALq+lmLp3gJBAAYvawPdGZz6TjiRpIJ4RbnQ== 2024-09-17T16:15:07.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQALq+lmLp3gJBAAYvawPdGZz6TjiRpIJ4RbnQ=="}' 2024-09-17T16:15:07.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1c0969d6-76b0-4bed-8825-36ef7650a841 -i td/test-erasure-eio/4/new.json 2024-09-17T16:15:08.541 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:15:08.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-17T16:15:08.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALq+lmLp3gJBAAYvawPdGZz6TjiRpIJ4RbnQ== --osd-uuid 1c0969d6-76b0-4bed-8825-36ef7650a841 2024-09-17T16:15:08.587 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:15:08.583+0000 7f8f5fc0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:15:08.588 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:15:08.585+0000 7f8f5fc0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:15:08.590 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:15:08.586+0000 7f8f5fc0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:15:08.590 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:15:08.587+0000 7f8f5fc0b640 -1 bdev(0x559359653c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-17T16:15:08.590 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:15:08.587+0000 7f8f5fc0b640 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-17T16:15:14.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-17T16:15:14.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:15:14.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-17T16:15:14.841 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-17T16:15:14.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:15:15.384 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-17T16:15:15.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-17T16:15:15.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:15:15.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:15:15.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:15:15.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:15:15.416 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:15:15.414+0000 7fad315c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:15:15.459 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:15:15.457+0000 7fad315c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:15:15.459 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:15:15.458+0000 7fad315c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:15:15.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-17T16:15:15.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:15:15.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-17T16:15:15.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:15:15.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:15:15.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:15.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:15:15.822 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:15:15.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:15.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:16.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:16.826 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:15:16.824+0000 7fad315c5640 -1 Falling back to public interface 2024-09-17T16:15:17.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:17.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:17.257 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:15:17.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:15:17.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:17.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:17.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:18.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:18.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:18.675 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:15:18.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:15:18.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:18.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:19.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:19.804 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:15:19.802+0000 7fad315c5640 -1 osd.4 0 log_to_monitors true 2024-09-17T16:15:20.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:20.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:20.110 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:15:20.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:15:20.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:20.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:20.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:21.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:21.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:21.674 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:15:21.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:15:21.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:21.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:22.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:23.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:23.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:23.525 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:15:23.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:15:23.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:23.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:24.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:25.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:25.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:25.150 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:15:25.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:15:25.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:25.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:25.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:26.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:26.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:26.632 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:15:26.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:15:26.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:26.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:27.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:28.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:28.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:28.075 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:15:28.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:15:28.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:28.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:28.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:29.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:29.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:29.586 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:15:29.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:15:29.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:29.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:30.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:31.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:31.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:31.102 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:15:31.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:15:31.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:31.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:31.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:32.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:32.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:32.632 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:15:32.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:15:32.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:32.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:34.255 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:15:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:15:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:35.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:36.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:36.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:36.459 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:15:36.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:15:36.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:36.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:37.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:38.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:38.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:38.320 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:15:38.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:15:38.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:38.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:39.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:40.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:40.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:40.080 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:15:40.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:15:40.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:40.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:40.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:41.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:41.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:41.974 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:15:41.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:15:41.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:41.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:42.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:15:43.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:15:43.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:15:43.719 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:15:43.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:15:43.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:15:43.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:15:44.196 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/4242127385,v1:127.0.0.1:6835/4242127385] [v2:127.0.0.1:6836/4242127385,v1:127.0.0.1:6837/4242127385] exists,up 1c0969d6-76b0-4bed-8825-36ef7650a841 2024-09-17T16:15:44.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:15:44.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:15:44.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:15:44.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-17T16:15:44.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:15:44.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:15:44.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:15:44.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:15:44.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:15:44.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T16:15:44.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-17T16:15:44.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.0.asok log flush 2024-09-17T16:15:44.278 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:15:44.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-17T16:15:44.289 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:12:55.678+0000 7fc9697a6640 0 load: jerasure load: lrc 2024-09-17T16:15:44.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:540: TEST_ec_backfill_unfound: CEPH_ARGS='--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:15:44.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:542: TEST_ec_backfill_unfound: local poolname=pool-jerasure 2024-09-17T16:15:44.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:543: TEST_ec_backfill_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-09-17T16:15:44.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:15:44.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-17T16:15:44.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-17T16:15:44.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-17T16:15:44.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-17T16:15:44.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-17T16:15:44.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-17T16:15:45.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-17T16:15:45.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-17T16:15:46.172 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-17T16:15:46.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:15:47.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-17T16:15:47.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:15:47.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:15:47.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:15:47.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:15:47.186 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:15:47.186 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:15:47.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:15:47.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:15:47.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:15:47.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:15:47.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:15:47.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:15:47.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:15:47.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:15:47.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:15:47.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:15:47.791 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:15:47.791 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:15:47.791 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:15:47.791 INFO:tasks.workunit.client.0.smithi162.stderr:4' 2024-09-17T16:15:47.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:15:47.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:15:47.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:15:47.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803812 2024-09-17T16:15:47.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803812 2024-09-17T16:15:47.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812' 2024-09-17T16:15:47.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:15:47.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:15:48.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574881 2024-09-17T16:15:48.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574881 2024-09-17T16:15:48.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-55834574881' 2024-09-17T16:15:48.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:15:48.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:15:48.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345943 2024-09-17T16:15:48.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345943 2024-09-17T16:15:48.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-55834574881 2-85899345943' 2024-09-17T16:15:48.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:15:48.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:15:48.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018894 2024-09-17T16:15:48.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018894 2024-09-17T16:15:48.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-55834574881 2-85899345943 3-128849018894' 2024-09-17T16:15:48.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:15:48.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:15:48.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659142 2024-09-17T16:15:48.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659142 2024-09-17T16:15:48.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-55834574881 2-85899345943 3-128849018894 4-176093659142' 2024-09-17T16:15:48.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:15:48.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803812 2024-09-17T16:15:48.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:15:48.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:15:48.401 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803812 2024-09-17T16:15:48.401 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:15:48.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803812 2024-09-17T16:15:48.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803812' 2024-09-17T16:15:48.402 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803812 2024-09-17T16:15:48.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:15:48.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803812 -lt 25769803812 2024-09-17T16:15:48.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:15:48.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574881 2024-09-17T16:15:48.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:15:48.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:15:48.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574881 2024-09-17T16:15:48.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:15:48.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574881 2024-09-17T16:15:48.803 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574881 2024-09-17T16:15:48.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574881' 2024-09-17T16:15:48.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:15:49.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574881 -lt 55834574881 2024-09-17T16:15:49.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:15:49.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345943 2024-09-17T16:15:49.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:15:49.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:15:49.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345943 2024-09-17T16:15:49.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:15:49.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345943 2024-09-17T16:15:49.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345943' 2024-09-17T16:15:49.224 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 85899345943 2024-09-17T16:15:49.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:15:49.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345943 -lt 85899345943 2024-09-17T16:15:49.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:15:49.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018894 2024-09-17T16:15:49.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:15:49.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:15:49.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018894 2024-09-17T16:15:49.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:15:49.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018894 2024-09-17T16:15:49.658 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 128849018894 2024-09-17T16:15:49.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018894' 2024-09-17T16:15:49.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:15:50.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018894 -lt 128849018894 2024-09-17T16:15:50.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:15:50.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-176093659142 2024-09-17T16:15:50.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:15:50.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:15:50.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-176093659142 2024-09-17T16:15:50.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:15:50.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659142 2024-09-17T16:15:50.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 176093659142' 2024-09-17T16:15:50.076 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 176093659142 2024-09-17T16:15:50.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:15:50.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659142 -lt 176093659142 2024-09-17T16:15:50.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:15:50.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:15:50.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:15:50.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:15:50.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:15:50.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:15:50.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:15:50.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:15:50.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:15:50.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:15:50.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:15:51.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:15:51.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:15:51.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:15:51.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:15:51.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:15:51.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:15:51.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:15:51.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:545: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-17T16:15:52.281 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T16:15:52.281 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:15:48.084073+0000 0'0 47:12 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-17T16:15:45.736175+0000 0'0 2024-09-17T16:15:45.736175+0000 0 0 periodic scrub scheduled @ 2024-09-18T17:02:53.392736+0000 0 0 2024-09-17T16:15:52.281 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:15:46.064489+0000 0'0 48:21 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-18T19:20:14.515168+0000 0 0 2024-09-17T16:15:52.281 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:15:45.837304+0000 0'0 47:113 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:51:06.394173+0000 0 0 2024-09-17T16:15:52.281 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:15:46.717882+0000 0'0 48:21 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-18T19:54:26.684807+0000 0 0 2024-09-17T16:15:52.281 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:14:32.684526+0000 0'0 47:72 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-19T04:04:15.095543+0000 0 0 2024-09-17T16:15:52.281 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-09-17T16:15:52.281 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T16:15:52.281 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-09-17T16:15:52.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:547: TEST_ec_backfill_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-17T16:15:52.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:15:52.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:15:52.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-17T16:15:52.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:15:52.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:15:52.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:15:52.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:15:52.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:15:52.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:15:52.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:15:52.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:15:52.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:52.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: get_primary pool-jerasure myobject 2024-09-17T16:15:52.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=pool-jerasure 2024-09-17T16:15:52.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=myobject 2024-09-17T16:15:52.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map pool-jerasure myobject 2024-09-17T16:15:52.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T16:15:52.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: local primary=3 2024-09-17T16:15:52.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: get_osds pool-jerasure myobject 2024-09-17T16:15:52.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:15:52.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-17T16:15:52.856 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-17T16:15:52.856 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:15:53.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:15:53.284 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:15:53.284 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:15:53.285 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:15:53.285 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:15:53.285 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2 2024-09-17T16:15:53.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-09-17T16:15:53.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: local -a initial_osds 2024-09-17T16:15:53.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: local last_osd=2 2024-09-17T16:15:53.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-17T16:15:53.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:15:53.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:15:53.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:15:53.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:15:53.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:15:53.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:15:53.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: ceph osd down 2 2024-09-17T16:15:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already down. 2024-09-17T16:15:54.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: ceph osd out 2 2024-09-17T16:15:55.097 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already out. 2024-09-17T16:15:55.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-17T16:15:55.479 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T16:15:55.479 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 1 0 0 0 0 4096 0 0 1 0 1 active+clean 2024-09-17T16:15:48.084073+0000 48'1 48:17 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-17T16:15:45.736175+0000 0'0 2024-09-17T16:15:45.736175+0000 0 0 periodic scrub scheduled @ 2024-09-18T20:28:44.587487+0000 0 0 2024-09-17T16:15:55.480 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-17T16:15:54.174077+0000 0'0 50:29 [4,1] 4 [4,1] 4 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-19T03:32:31.990415+0000 0 0 2024-09-17T16:15:55.480 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:15:45.837304+0000 0'0 48:115 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:51:06.394173+0000 0 0 2024-09-17T16:15:55.480 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:15:46.717882+0000 0'0 50:25 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-18T19:54:26.684807+0000 0 0 2024-09-17T16:15:55.480 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:14:32.684526+0000 0'0 48:74 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-19T04:04:15.095543+0000 0 0 2024-09-17T16:15:55.480 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-09-17T16:15:55.480 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T16:15:55.480 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-09-17T16:15:55.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-09-17T16:15:55.492 INFO:tasks.workunit.client.0.smithi162.stderr:4+0 records in 2024-09-17T16:15:55.492 INFO:tasks.workunit.client.0.smithi162.stderr:4+0 records out 2024-09-17T16:15:55.492 INFO:tasks.workunit.client.0.smithi162.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 8.7199e-05 s, 47.0 MB/s 2024-09-17T16:15:55.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: seq 1 300 2024-09-17T16:15:55.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:55.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:55.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:55.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:56.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:56.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:56.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:56.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:56.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:56.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:57.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:57.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:57.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:57.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:57.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:57.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:57.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:57.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:58.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:58.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:58.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:58.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:58.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:58.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:59.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:59.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:59.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:59.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:59.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:59.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-09-17T16:15:59.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:15:59.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:00.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:00.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:00.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:00.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:00.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:00.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:00.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:00.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:01.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:01.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:01.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:01.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:01.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:01.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:01.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:01.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:02.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:02.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:02.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:02.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:02.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:02.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:02.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:02.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:03.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:03.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:03.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:03.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:03.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:03.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:03.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:03.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:04.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:04.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:04.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:04.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:04.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:04.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:04.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:04.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:05.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:05.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:05.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:05.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:05.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:05.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:05.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:05.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:06.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:06.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:06.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:06.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:06.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:06.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:06.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:06.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:07.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:07.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:07.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:07.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:07.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:07.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:07.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:07.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:08.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:08.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:08.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:08.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:08.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:08.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:09.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:09.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:09.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:09.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:09.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:09.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:09.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:09.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:10.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:10.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:10.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:10.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:10.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:10.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:10.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:10.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:11.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:11.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:11.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:11.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:11.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:11.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:11.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:11.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:12.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:12.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:12.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:12.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:12.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:12.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:12.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:12.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:13.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:13.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:13.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:13.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:13.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:13.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:13.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:13.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:14.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:14.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:14.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:14.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:14.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:14.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:14.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:14.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:14.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:14.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:15.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:15.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:15.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:15.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:15.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:15.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:15.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:15.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:16.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:16.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:16.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:16.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:16.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:16.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:16.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:16.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:17.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:17.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:17.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:17.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:17.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:17.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:17.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:17.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:18.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:18.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:18.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:18.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:18.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:18.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:18.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:18.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:19.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:19.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:19.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:19.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:19.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:19.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:19.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:19.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:20.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:20.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:20.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:20.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj101 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:20.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj102 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:20.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:20.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj103 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:21.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:21.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj104 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:21.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:21.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj105 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:21.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:21.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj106 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:21.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:21.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj107 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:22.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:22.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj108 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:22.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:22.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj109 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:22.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:22.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj110 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:22.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:22.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj111 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:23.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:23.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj112 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:23.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:23.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj113 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:23.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:23.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj114 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:24.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:24.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj115 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:24.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:24.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj116 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:24.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:24.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj117 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:24.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:24.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj118 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:25.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:25.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj119 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:25.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:25.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj120 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:25.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:25.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj121 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:25.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:25.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj122 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:26.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:26.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj123 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:26.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:26.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj124 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:26.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:26.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj125 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:26.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:26.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj126 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:26.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:26.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj127 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:27.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:27.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj128 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:27.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:27.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj129 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:27.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:27.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj130 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:27.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:27.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj131 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:28.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:28.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj132 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:28.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:28.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj133 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:28.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:28.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj134 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:28.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:28.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj135 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:29.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:29.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj136 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:29.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:29.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj137 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:29.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:29.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj138 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:29.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:29.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj139 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:30.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:30.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj140 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:30.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:30.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj141 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:30.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:30.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj142 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:30.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:30.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj143 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:31.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:31.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj144 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:31.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:31.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj145 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:31.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:31.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj146 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:31.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:31.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj147 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:32.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:32.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj148 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:32.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:32.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj149 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:32.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:32.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj150 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:32.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:32.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj151 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:32.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:32.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj152 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:33.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:33.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj153 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:33.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:33.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj154 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:33.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:33.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj155 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:33.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:33.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj156 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:34.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:34.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj157 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:34.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:34.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj158 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:34.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:34.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj159 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:34.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:34.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj160 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:35.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:35.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj161 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:35.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:35.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj162 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:35.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:35.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj163 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:35.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:35.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj164 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:35.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:35.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj165 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:36.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:36.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj166 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:36.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:36.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj167 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:36.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:36.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj168 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:36.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:36.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj169 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:37.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:37.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj170 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:37.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:37.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj171 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:37.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:37.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj172 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:37.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:37.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj173 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:38.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:38.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj174 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:38.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:38.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj175 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:38.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:38.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj176 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:38.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:38.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj177 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:39.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:39.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj178 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:39.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:39.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj179 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:39.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:39.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj180 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:39.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:39.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj181 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:40.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:40.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj182 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:40.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:40.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj183 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:40.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:40.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj184 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:40.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:40.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj185 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:41.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:41.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj186 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:41.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:41.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj187 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:41.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:41.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj188 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:41.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:41.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj189 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:42.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:42.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj190 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:42.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:42.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj191 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:42.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:42.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj192 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:42.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:42.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj193 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:43.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:43.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj194 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj195 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:43.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:43.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj196 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:43.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:43.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj197 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:44.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:44.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj198 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:44.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:44.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj199 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:44.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:44.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj200 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:44.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:44.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj201 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:45.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:45.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj202 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:45.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:45.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj203 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:45.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:45.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj204 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:45.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:45.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj205 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:45.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:45.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj206 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:46.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:46.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj207 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:46.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:46.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj208 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:46.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:46.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj209 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:46.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:46.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj210 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:47.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:47.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj211 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:47.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:47.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj212 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:47.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:47.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj213 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:48.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:48.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj214 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:48.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:48.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj215 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:48.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:48.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj216 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:48.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:48.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj217 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:49.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:49.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj218 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:49.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:49.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj219 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:49.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:49.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj220 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:49.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:49.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj221 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:49.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:49.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj222 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:50.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:50.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj223 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:50.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:50.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj224 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:50.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:50.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj225 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:50.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:50.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj226 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj227 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:51.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:51.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj228 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:51.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:51.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj229 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:51.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:51.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj230 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:52.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:52.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj231 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:52.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:52.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj232 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:52.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:52.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj233 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:52.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:52.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj234 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:53.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:53.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj235 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:53.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:53.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj236 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:53.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:53.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj237 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:53.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:53.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj238 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:54.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:54.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj239 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:54.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:54.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj240 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:54.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:54.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj241 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:55.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:55.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj242 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:55.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:55.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj243 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:55.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:55.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj244 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:55.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:55.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj245 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:55.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:55.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj246 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:56.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:56.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj247 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:56.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:56.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj248 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:56.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:56.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj249 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:56.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:56.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj250 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:57.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:57.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj251 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:57.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:57.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj252 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:57.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:57.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj253 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:57.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:57.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj254 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:58.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:58.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj255 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:58.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:58.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj256 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:58.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:58.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj257 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:58.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:58.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj258 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:58.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:58.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj259 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj260 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj261 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:59.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:59.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj262 td/test-erasure-eio/ORIGINAL 2024-09-17T16:16:59.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:16:59.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj263 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:00.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:00.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj264 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:00.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:00.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj265 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:00.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:00.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj266 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:00.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:00.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj267 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:01.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:01.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj268 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:01.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:01.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj269 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:01.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:01.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj270 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:01.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:01.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj271 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:02.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:02.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj272 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:02.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:02.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj273 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:02.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:02.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj274 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:02.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:02.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj275 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:03.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:03.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj276 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:03.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:03.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj277 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:03.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:03.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj278 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:03.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:03.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj279 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:04.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:04.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj280 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:04.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:04.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj281 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:04.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:04.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj282 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:04.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:04.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj283 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:04.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:04.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj284 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:05.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:05.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj285 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:05.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:05.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj286 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:05.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:05.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj287 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:05.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:05.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj288 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj289 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj290 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:06.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:06.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj291 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:06.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:06.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj292 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:07.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:07.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj293 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:07.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:07.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj294 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:07.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:07.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj295 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:07.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:07.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj296 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:08.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:08.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj297 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:08.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:08.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj298 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:08.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:08.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj299 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:08.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:17:08.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj300 td/test-erasure-eio/ORIGINAL 2024-09-17T16:17:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:564: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 0 2024-09-17T16:17:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T16:17:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T16:17:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T16:17:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T16:17:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T16:17:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T16:17:09.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj250 2024-09-17T16:17:09.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T16:17:09.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T16:17:09.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T16:17:09.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-17T16:17:09.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T16:17:09.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj250 2024-09-17T16:17:09.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:17:09.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj250 2024-09-17T16:17:09.067 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj250 2024-09-17T16:17:09.068 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:17:09.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:17:09.502 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:17:09.502 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:17:09.502 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:17:09.502 INFO:tasks.workunit.client.0.smithi162.stderr:2147483647' 2024-09-17T16:17:09.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2024-09-17T16:17:09.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-17T16:17:09.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T16:17:09.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-17T16:17:09.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T16:17:09.503 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-17T16:17:09.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T16:17:09.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-17T16:17:09.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T16:17:09.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-17T16:17:09.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T16:17:09.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T16:17:09.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T16:17:09.505 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-17T16:17:09.505 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-17T16:17:09.505 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-17T16:17:09.505 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:17:09.505 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:17:09.505 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:17:09.505 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.3.asok 2024-09-17T16:17:09.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-17T16:17:09.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T16:17:09.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T16:17:09.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T16:17:09.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-17T16:17:09.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-17T16:17:09.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-17T16:17:09.592 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:17:09.592 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:17:09.592 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:17:09.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.3.asok 2024-09-17T16:17:09.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T16:17:09.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.3.asok injectdataerr pool-jerasure obj250 0 2024-09-17T16:17:09.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:565: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 1 2024-09-17T16:17:09.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T16:17:09.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T16:17:09.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T16:17:09.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T16:17:09.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T16:17:09.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T16:17:09.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj250 2024-09-17T16:17:09.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T16:17:09.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T16:17:09.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T16:17:09.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-17T16:17:09.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T16:17:09.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj250 2024-09-17T16:17:09.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:17:09.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj250 2024-09-17T16:17:09.683 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj250 2024-09-17T16:17:09.683 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:17:10.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:17:10.103 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:17:10.103 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:17:10.103 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:17:10.103 INFO:tasks.workunit.client.0.smithi162.stderr:2147483647' 2024-09-17T16:17:10.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2024-09-17T16:17:10.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-17T16:17:10.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T16:17:10.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-17T16:17:10.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T16:17:10.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/1/type 2024-09-17T16:17:10.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T16:17:10.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-17T16:17:10.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T16:17:10.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-17T16:17:10.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T16:17:10.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T16:17:10.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T16:17:10.106 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-17T16:17:10.106 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T16:17:10.106 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T16:17:10.106 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:17:10.106 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:17:10.106 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:17:10.106 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.1.asok 2024-09-17T16:17:10.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-17T16:17:10.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T16:17:10.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T16:17:10.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T16:17:10.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-17T16:17:10.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T16:17:10.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T16:17:10.192 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:17:10.192 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:17:10.192 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:17:10.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.1.asok 2024-09-17T16:17:10.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T16:17:10.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.1.asok injectdataerr pool-jerasure obj250 1 2024-09-17T16:17:10.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:567: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-09-17T16:17:10.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:17:10.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:17:10.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T16:17:10.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:17:10.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:17:10.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:17:10.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:17:10.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:17:10.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:17:10.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:17:10.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:17:10.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:17:10.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:17:10.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:17:10.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:17:10.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:17:10.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:17:10.283 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:17:10.283 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:17:10.283 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:17:10.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:17:10.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:17:10.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:17:10.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:17:10.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:17:10.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:17:10.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:17:10.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:17:10.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:17:10.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:17:10.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:17:10.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:17:10.284 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T16:17:10.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T16:17:10.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:17:10.285 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-17T16:17:10.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T16:17:10.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:17:10.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:17:10.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:17:10.307 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:17:10.306+0000 7f18011a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:17:10.309 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:17:10.307+0000 7f18011a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:17:10.310 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:17:10.308+0000 7f18011a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:17:10.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T16:17:10.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:17:10.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:17:10.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:17:10.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:17:10.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:17:10.672 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:17:10.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:17:10.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:17:10.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:17:11.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:17:11.409 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:17:11.408+0000 7f18011a8640 -1 Falling back to public interface 2024-09-17T16:17:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:17:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:17:12.075 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:17:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:17:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:17:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:17:12.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:17:13.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:17:13.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:17:13.500 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:17:13.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:17:13.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:17:13.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:17:13.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:17:14.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:17:14.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:17:14.969 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:17:14.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:17:14.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:17:14.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:17:14.972 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:17:14.970+0000 7f18011a8640 -1 osd.2 48 log_to_monitors true 2024-09-17T16:17:15.432 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up out weight 0 up_from 56 up_thru 22 down_at 49 last_clean_interval [20,48) [v2:127.0.0.1:6818/3781281694,v1:127.0.0.1:6819/3781281694] [v2:127.0.0.1:6820/3781281694,v1:127.0.0.1:6821/3781281694] exists,up d09e1f7c-4713-45c7-9eb5-9f689f2aca77 2024-09-17T16:17:15.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:17:15.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:17:15.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:17:15.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:568: TEST_ec_backfill_unfound: ceph osd in 2 2024-09-17T16:17:16.655 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already in. 2024-09-17T16:17:16.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: sleep 15 2024-09-17T16:17:31.669 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: seq 1 240 2024-09-17T16:17:31.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-17T16:17:31.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-17T16:17:31.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-17T16:17:31.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-17T16:17:31.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-17T16:17:31.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-17T16:17:32.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:32.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:32.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-17T16:17:32.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-17T16:17:32.066 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:32.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:32.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-17T16:17:33.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-17T16:17:33.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-17T16:17:33.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-17T16:17:33.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-17T16:17:33.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-17T16:17:33.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-17T16:17:33.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:33.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:33.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-17T16:17:33.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-17T16:17:33.464 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:33.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:33.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-17T16:17:34.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-17T16:17:34.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-17T16:17:34.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-17T16:17:34.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-17T16:17:34.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-17T16:17:34.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-17T16:17:34.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:34.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:34.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-17T16:17:34.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-17T16:17:34.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:34.852 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:34.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-17T16:17:35.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-17T16:17:35.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-17T16:17:35.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-17T16:17:35.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-17T16:17:35.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-17T16:17:35.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-17T16:17:36.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:36.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:36.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-17T16:17:36.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-17T16:17:36.234 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:36.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:36.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-17T16:17:37.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-17T16:17:37.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-17T16:17:37.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-17T16:17:37.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-17T16:17:37.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-17T16:17:37.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-17T16:17:37.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:37.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:37.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-17T16:17:37.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-17T16:17:37.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:37.630 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:37.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-17T16:17:38.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-17T16:17:38.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-17T16:17:38.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-17T16:17:38.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-17T16:17:38.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-17T16:17:38.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-17T16:17:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-17T16:17:39.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-17T16:17:39.021 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:39.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:39.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-17T16:17:40.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-17T16:17:40.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-17T16:17:40.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-17T16:17:40.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-17T16:17:40.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-17T16:17:40.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-17T16:17:40.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:40.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:40.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-17T16:17:40.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-17T16:17:40.412 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:40.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:40.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-17T16:17:41.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-17T16:17:41.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-17T16:17:41.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-17T16:17:41.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-17T16:17:41.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-17T16:17:41.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-17T16:17:41.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:41.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:41.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-17T16:17:41.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-17T16:17:41.801 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:41.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:41.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-17T16:17:42.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-17T16:17:42.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-17T16:17:42.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-17T16:17:42.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-17T16:17:42.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-17T16:17:42.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-17T16:17:43.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:43.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:43.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-17T16:17:43.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-17T16:17:43.190 INFO:tasks.workunit.client.0.smithi162.stdout:active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:43.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-17T16:17:43.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-17T16:17:44.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-17T16:17:44.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-17T16:17:44.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-17T16:17:44.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-17T16:17:44.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-17T16:17:44.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-17T16:17:44.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-09-17T16:17:44.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-09-17T16:17:44.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-17T16:17:44.578 INFO:tasks.workunit.client.0.smithi162.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-09-17T16:17:44.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 0 = 0 ']' 2024-09-17T16:17:44.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:576: TEST_ec_backfill_unfound: break 2024-09-17T16:17:44.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:582: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-17T16:17:44.999 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T16:17:44.999 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-09-17T16:17:38.386322+0000 54'301 61:901 [3,1,4,0,2] 3 [3,1,4,0,NONE] 3 0'0 2024-09-17T16:15:45.736175+0000 0'0 2024-09-17T16:15:45.736175+0000 0 0 no scrub is scheduled 0 0 2024-09-17T16:17:44.999 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:17:17.376965+0000 0'0 61:62 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:47:38.225631+0000 0 0 2024-09-17T16:17:44.999 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:15:45.837304+0000 0'0 61:140 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:51:06.394173+0000 0 0 2024-09-17T16:17:44.999 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:15:46.717882+0000 0'0 61:45 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-18T19:54:26.684807+0000 0 0 2024-09-17T16:17:44.999 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:17:17.417117+0000 0'0 61:115 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:32:10.859575+0000 0 0 2024-09-17T16:17:44.999 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-09-17T16:17:44.999 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T16:17:44.999 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-09-17T16:17:45.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:583: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-17T16:17:45.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:17:45.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:17:45.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:17:45.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:17:45.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:17:45.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:17:45.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:584: TEST_ec_backfill_unfound: sleep 5 2024-09-17T16:17:50.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:586: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-17T16:17:50.697 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T16:17:50.697 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 301 1 303 0 1 1232896 0 0 101 200 101 active+recovery_unfound+undersized+degraded 2024-09-17T16:17:46.480324+0000 54'301 64:915 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-09-17T16:15:45.736175+0000 0'0 2024-09-17T16:15:45.736175+0000 0 0 no scrub is scheduled 0 0 2024-09-17T16:17:50.697 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-17T16:17:46.240081+0000 0'0 64:71 [4,1] 4 [4,1] 4 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:04:42.077214+0000 0 0 2024-09-17T16:17:50.697 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:15:45.837304+0000 0'0 63:144 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:51:06.394173+0000 0 0 2024-09-17T16:17:50.697 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:15:46.717882+0000 0'0 64:51 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-18T19:54:26.684807+0000 0 0 2024-09-17T16:17:50.697 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-17T16:17:45.870730+0000 0'0 64:125 [1,0] 1 [1,0] 1 0'0 2024-09-17T16:12:46.675005+0000 0'0 2024-09-17T16:12:46.675005+0000 0 0 periodic scrub scheduled @ 2024-09-18T20:48:36.971383+0000 0 0 2024-09-17T16:17:50.697 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-09-17T16:17:50.697 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T16:17:50.697 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-09-17T16:17:50.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:587: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-17T16:17:50.817 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-09-17T16:17:50.817 INFO:tasks.workunit.client.0.smithi162.stdout: "num_missing": 1, 2024-09-17T16:17:50.817 INFO:tasks.workunit.client.0.smithi162.stdout: "num_unfound": 1, 2024-09-17T16:17:50.817 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": [ 2024-09-17T16:17:50.817 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.817 INFO:tasks.workunit.client.0.smithi162.stdout: "oid": { 2024-09-17T16:17:50.817 INFO:tasks.workunit.client.0.smithi162.stdout: "oid": "obj250", 2024-09-17T16:17:50.817 INFO:tasks.workunit.client.0.smithi162.stdout: "key": "", 2024-09-17T16:17:50.818 INFO:tasks.workunit.client.0.smithi162.stdout: "snapid": -2, 2024-09-17T16:17:50.818 INFO:tasks.workunit.client.0.smithi162.stdout: "hash": 2249616407, 2024-09-17T16:17:50.818 INFO:tasks.workunit.client.0.smithi162.stdout: "max": 0, 2024-09-17T16:17:50.818 INFO:tasks.workunit.client.0.smithi162.stdout: "pool": 2, 2024-09-17T16:17:50.818 INFO:tasks.workunit.client.0.smithi162.stdout: "namespace": "" 2024-09-17T16:17:50.818 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.818 INFO:tasks.workunit.client.0.smithi162.stdout: "need": "54'251", 2024-09-17T16:17:50.818 INFO:tasks.workunit.client.0.smithi162.stdout: "have": "0'0", 2024-09-17T16:17:50.818 INFO:tasks.workunit.client.0.smithi162.stdout: "flags": "none", 2024-09-17T16:17:50.818 INFO:tasks.workunit.client.0.smithi162.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-17T16:17:50.818 INFO:tasks.workunit.client.0.smithi162.stdout: "locations": [ 2024-09-17T16:17:50.818 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-17T16:17:50.818 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-17T16:17:50.818 INFO:tasks.workunit.client.0.smithi162.stdout: ] 2024-09-17T16:17:50.819 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:17:50.819 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.819 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "NotRecovering", 2024-09-17T16:17:50.819 INFO:tasks.workunit.client.0.smithi162.stdout: "available_might_have_unfound": true, 2024-09-17T16:17:50.819 INFO:tasks.workunit.client.0.smithi162.stdout: "might_have_unfound": [ 2024-09-17T16:17:50.819 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.819 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "2(4)", 2024-09-17T16:17:50.819 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "osd is down" 2024-09-17T16:17:50.819 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:17:50.819 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.819 INFO:tasks.workunit.client.0.smithi162.stdout: "more": false 2024-09-17T16:17:50.819 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-09-17T16:17:50.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:588: TEST_ec_backfill_unfound: ceph pg 2.0 query 2024-09-17T16:17:50.936 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-09-17T16:17:50.937 INFO:tasks.workunit.client.0.smithi162.stdout: "snap_trimq": "[]", 2024-09-17T16:17:50.937 INFO:tasks.workunit.client.0.smithi162.stdout: "snap_trimq_len": 0, 2024-09-17T16:17:50.937 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-09-17T16:17:50.937 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch": 64, 2024-09-17T16:17:50.937 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-17T16:17:50.937 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:17:50.937 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:17:50.937 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:17:50.937 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:17:50.937 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-17T16:17:50.937 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.937 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-17T16:17:50.938 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:17:50.938 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:17:50.938 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:17:50.938 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:17:50.938 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-17T16:17:50.938 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.938 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_recovery_backfill": [ 2024-09-17T16:17:50.938 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-17T16:17:50.938 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-17T16:17:50.938 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-17T16:17:50.938 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-17T16:17:50.938 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.938 INFO:tasks.workunit.client.0.smithi162.stdout: "info": { 2024-09-17T16:17:50.938 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s0", 2024-09-17T16:17:50.939 INFO:tasks.workunit.client.0.smithi162.stdout: "shared": "0", 2024-09-17T16:17:50.939 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "54'301", 2024-09-17T16:17:50.939 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "54'250", 2024-09-17T16:17:50.939 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "54'200", 2024-09-17T16:17:50.939 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 301, 2024-09-17T16:17:50.939 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-17T16:17:50.939 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-17T16:17:50.939 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-17T16:17:50.939 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 46, 2024-09-17T16:17:50.939 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 46, 2024-09-17T16:17:50.939 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 64, 2024-09-17T16:17:50.939 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 62, 2024-09-17T16:17:50.939 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:17:50.940 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 46, 2024-09-17T16:17:50.940 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-17T16:17:50.940 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-17T16:17:50.940 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 62, 2024-09-17T16:17:50.940 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 62, 2024-09-17T16:17:50.940 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 46, 2024-09-17T16:17:50.940 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:17:50.940 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.940 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:17:50.940 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.940 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.940 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-17T16:17:50.941 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.941 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-17T16:17:50.941 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "54'301", 2024-09-17T16:17:50.941 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 916, 2024-09-17T16:17:50.941 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 64, 2024-09-17T16:17:50.941 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-09-17T16:17:50.941 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-17T16:17:47.783592+0000", 2024-09-17T16:17:50.941 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-17T16:17:46.480324+0000", 2024-09-17T16:17:50.941 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-17T16:17:47.783592+0000", 2024-09-17T16:17:50.941 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-17T16:17:47.783592+0000", 2024-09-17T16:17:50.941 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-17T16:15:52.464689+0000", 2024-09-17T16:17:50.941 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-17T16:17:46.467133+0000", 2024-09-17T16:17:50.941 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-17T16:17:46.467133+0000", 2024-09-17T16:17:50.942 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-17T16:17:47.783592+0000", 2024-09-17T16:17:50.942 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-17T16:17:46.148012+0000", 2024-09-17T16:17:50.942 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-17T16:17:46.147703+0000", 2024-09-17T16:17:50.942 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 62, 2024-09-17T16:17:50.942 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "54'200", 2024-09-17T16:17:50.942 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "54'200", 2024-09-17T16:17:50.942 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 46, 2024-09-17T16:17:50.942 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:17:50.942 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-17T16:17:50.942 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-17T16:17:50.942 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:17:50.942 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.942 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:17:50.942 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.943 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.943 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-17T16:17:50.943 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-17T16:17:50.943 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 200, 2024-09-17T16:17:50.943 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-17T16:17:50.943 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-17T16:17:50.943 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-17T16:17:50.943 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-17T16:17:50.943 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-17T16:17:50.943 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-17T16:17:50.943 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-17T16:17:50.943 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-17T16:17:50.944 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-17T16:17:50.944 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-17T16:17:50.944 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "no scrub is scheduled", 2024-09-17T16:17:50.944 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-17T16:17:50.944 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-17T16:17:50.944 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-17T16:17:50.944 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-17T16:17:50.944 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 1232896, 2024-09-17T16:17:50.944 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 301, 2024-09-17T16:17:50.944 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-17T16:17:50.944 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 1505, 2024-09-17T16:17:50.944 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 1, 2024-09-17T16:17:50.945 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 1, 2024-09-17T16:17:50.945 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 303, 2024-09-17T16:17:50.945 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-17T16:17:50.945 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 1, 2024-09-17T16:17:50.945 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 301, 2024-09-17T16:17:50.945 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-17T16:17:50.945 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-17T16:17:50.945 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-17T16:17:50.945 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 301, 2024-09-17T16:17:50.946 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 1204, 2024-09-17T16:17:50.946 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-17T16:17:50.946 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T16:17:50.946 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-17T16:17:50.946 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-17T16:17:50.946 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-17T16:17:50.946 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-17T16:17:50.946 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-17T16:17:50.946 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T16:17:50.946 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T16:17:50.946 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-17T16:17:50.946 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-17T16:17:50.946 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-17T16:17:50.947 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-17T16:17:50.947 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-17T16:17:50.947 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-17T16:17:50.947 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-17T16:17:50.947 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-17T16:17:50.947 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-17T16:17:50.947 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-17T16:17:50.947 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-17T16:17:50.947 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-17T16:17:50.947 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-17T16:17:50.947 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-17T16:17:50.947 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-17T16:17:50.948 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-17T16:17:50.948 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.948 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-17T16:17:50.948 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:17:50.948 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:17:50.948 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:17:50.948 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:17:50.948 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-17T16:17:50.948 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.948 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-17T16:17:50.948 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:17:50.948 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:17:50.948 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:17:50.949 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:17:50.949 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-17T16:17:50.949 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.949 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-17T16:17:50.949 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-17T16:17:50.949 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-17T16:17:50.949 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.949 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-17T16:17:50.949 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.949 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-17T16:17:50.949 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 300 2024-09-17T16:17:50.949 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.949 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.950 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),4(2)", 2024-09-17T16:17:50.950 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 1 2024-09-17T16:17:50.950 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:17:50.950 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.950 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-17T16:17:50.950 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-17T16:17:50.950 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-17T16:17:50.950 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-17T16:17:50.950 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.950 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-17T16:17:50.950 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-17T16:17:50.950 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-17T16:17:50.950 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 64, 2024-09-17T16:17:50.950 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-17T16:17:50.951 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-17T16:17:50.951 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-17T16:17:50.951 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:17:50.951 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.951 INFO:tasks.workunit.client.0.smithi162.stdout: "peer_info": [ 2024-09-17T16:17:50.951 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.951 INFO:tasks.workunit.client.0.smithi162.stdout: "peer": "0(3)", 2024-09-17T16:17:50.951 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s3", 2024-09-17T16:17:50.951 INFO:tasks.workunit.client.0.smithi162.stdout: "shared": "3", 2024-09-17T16:17:50.951 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "54'301", 2024-09-17T16:17:50.951 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "54'301", 2024-09-17T16:17:50.951 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "54'200", 2024-09-17T16:17:50.951 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 301, 2024-09-17T16:17:50.951 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-17T16:17:50.952 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-17T16:17:50.952 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-17T16:17:50.952 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 46, 2024-09-17T16:17:50.952 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 46, 2024-09-17T16:17:50.952 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 64, 2024-09-17T16:17:50.952 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 62, 2024-09-17T16:17:50.952 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:17:50.952 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 46, 2024-09-17T16:17:50.952 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-17T16:17:50.952 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-17T16:17:50.952 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 62, 2024-09-17T16:17:50.952 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 62, 2024-09-17T16:17:50.953 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 46, 2024-09-17T16:17:50.953 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:17:50.953 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.953 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:17:50.953 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.953 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.953 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-17T16:17:50.953 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.953 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-17T16:17:50.953 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "54'301", 2024-09-17T16:17:50.953 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 904, 2024-09-17T16:17:50.953 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 62, 2024-09-17T16:17:50.953 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+undersized+degraded", 2024-09-17T16:17:50.953 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-17T16:17:09.061093+0000", 2024-09-17T16:17:50.954 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-17T16:15:54.716937+0000", 2024-09-17T16:17:50.954 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-17T16:17:09.061093+0000", 2024-09-17T16:17:50.954 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-17T16:17:09.061093+0000", 2024-09-17T16:17:50.954 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-17T16:15:52.464689+0000", 2024-09-17T16:17:50.954 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-17T16:15:54.716937+0000", 2024-09-17T16:17:50.954 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-17T16:15:54.716937+0000", 2024-09-17T16:17:50.954 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-17T16:17:09.061093+0000", 2024-09-17T16:17:50.954 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-17T16:15:54.375192+0000", 2024-09-17T16:17:50.954 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-17T16:15:54.374952+0000", 2024-09-17T16:17:50.954 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 62, 2024-09-17T16:17:50.954 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "54'200", 2024-09-17T16:17:50.954 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "54'200", 2024-09-17T16:17:50.954 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 46, 2024-09-17T16:17:50.954 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:17:50.955 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-17T16:17:50.955 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-17T16:17:50.955 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:17:50.955 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.955 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:17:50.955 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.955 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.955 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-17T16:17:50.955 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-17T16:17:50.955 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 200, 2024-09-17T16:17:50.955 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-17T16:17:50.955 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-17T16:17:50.955 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-17T16:17:50.955 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-17T16:17:50.956 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-17T16:17:50.956 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-17T16:17:50.956 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-17T16:17:50.956 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-17T16:17:50.956 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-17T16:17:50.956 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-17T16:17:50.956 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-18T23:50:50.279195+0000", 2024-09-17T16:17:50.956 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-17T16:17:50.956 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-17T16:17:50.956 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-17T16:17:50.956 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-17T16:17:50.956 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 1232896, 2024-09-17T16:17:50.956 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 301, 2024-09-17T16:17:50.957 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-17T16:17:50.957 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 1505, 2024-09-17T16:17:50.957 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T16:17:50.957 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 0, 2024-09-17T16:17:50.957 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 301, 2024-09-17T16:17:50.957 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-17T16:17:50.957 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-09-17T16:17:50.957 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 301, 2024-09-17T16:17:50.957 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-17T16:17:50.957 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-17T16:17:50.957 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-17T16:17:50.957 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 301, 2024-09-17T16:17:50.957 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 1204, 2024-09-17T16:17:50.958 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-17T16:17:50.958 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T16:17:50.958 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-17T16:17:50.958 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-17T16:17:50.958 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-17T16:17:50.958 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-17T16:17:50.958 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-17T16:17:50.958 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T16:17:50.958 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T16:17:50.958 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-17T16:17:50.958 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-17T16:17:50.958 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-17T16:17:50.958 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-17T16:17:50.958 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-17T16:17:50.959 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-17T16:17:50.959 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-17T16:17:50.959 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-17T16:17:50.959 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-17T16:17:50.959 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-17T16:17:50.959 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-17T16:17:50.959 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-17T16:17:50.959 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-17T16:17:50.959 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-17T16:17:50.959 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-17T16:17:50.959 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-17T16:17:50.959 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.959 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-17T16:17:50.959 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:17:50.960 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:17:50.960 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:17:50.960 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:17:50.960 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-17T16:17:50.960 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.960 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-17T16:17:50.960 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:17:50.960 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:17:50.960 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:17:50.960 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:17:50.960 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-17T16:17:50.960 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.960 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-17T16:17:50.960 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-17T16:17:50.960 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-17T16:17:50.961 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-17T16:17:50.961 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-17T16:17:50.961 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.961 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-17T16:17:50.961 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.961 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-17T16:17:50.961 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 301 2024-09-17T16:17:50.961 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:17:50.961 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.961 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-17T16:17:50.961 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-17T16:17:50.961 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-17T16:17:50.961 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-17T16:17:50.962 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.962 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-17T16:17:50.962 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-17T16:17:50.962 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-17T16:17:50.962 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 64, 2024-09-17T16:17:50.962 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-17T16:17:50.962 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-17T16:17:50.962 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-17T16:17:50.962 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:17:50.962 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.962 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.962 INFO:tasks.workunit.client.0.smithi162.stdout: "peer": "1(1)", 2024-09-17T16:17:50.962 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s1", 2024-09-17T16:17:50.963 INFO:tasks.workunit.client.0.smithi162.stdout: "shared": "1", 2024-09-17T16:17:50.963 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "54'301", 2024-09-17T16:17:50.963 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "54'301", 2024-09-17T16:17:50.963 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "54'200", 2024-09-17T16:17:50.963 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 301, 2024-09-17T16:17:50.963 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-17T16:17:50.963 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-17T16:17:50.963 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-17T16:17:50.963 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 46, 2024-09-17T16:17:50.963 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 46, 2024-09-17T16:17:50.963 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 64, 2024-09-17T16:17:50.963 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 62, 2024-09-17T16:17:50.963 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:17:50.963 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 46, 2024-09-17T16:17:50.964 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-17T16:17:50.964 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-17T16:17:50.964 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 62, 2024-09-17T16:17:50.964 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 62, 2024-09-17T16:17:50.964 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 46, 2024-09-17T16:17:50.964 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:17:50.964 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.964 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:17:50.964 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.964 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.964 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-17T16:17:50.964 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.964 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-17T16:17:50.964 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "54'301", 2024-09-17T16:17:50.965 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 335, 2024-09-17T16:17:50.965 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 54, 2024-09-17T16:17:50.965 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+undersized+degraded", 2024-09-17T16:17:50.965 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-17T16:17:09.061093+0000", 2024-09-17T16:17:50.965 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-17T16:15:54.716937+0000", 2024-09-17T16:17:50.965 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-17T16:17:09.061093+0000", 2024-09-17T16:17:50.965 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-17T16:17:09.061093+0000", 2024-09-17T16:17:50.965 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-17T16:15:52.464689+0000", 2024-09-17T16:17:50.965 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-17T16:15:54.716937+0000", 2024-09-17T16:17:50.965 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-17T16:15:54.716937+0000", 2024-09-17T16:17:50.965 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-17T16:17:09.061093+0000", 2024-09-17T16:17:50.965 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-17T16:15:54.375192+0000", 2024-09-17T16:17:50.965 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-17T16:15:54.374952+0000", 2024-09-17T16:17:50.966 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 62, 2024-09-17T16:17:50.966 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "54'200", 2024-09-17T16:17:50.966 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "54'200", 2024-09-17T16:17:50.966 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 46, 2024-09-17T16:17:50.966 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:17:50.966 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-17T16:17:50.966 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-17T16:17:50.966 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:17:50.966 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.966 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:17:50.966 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.966 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.966 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-17T16:17:50.966 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-17T16:17:50.966 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 200, 2024-09-17T16:17:50.967 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-17T16:17:50.967 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-17T16:17:50.967 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-17T16:17:50.967 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-17T16:17:50.967 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-17T16:17:50.967 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-17T16:17:50.967 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-17T16:17:50.967 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-17T16:17:50.967 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-17T16:17:50.967 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-17T16:17:50.967 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-18T23:50:50.279195+0000", 2024-09-17T16:17:50.967 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-17T16:17:50.967 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-17T16:17:50.968 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-17T16:17:50.968 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-17T16:17:50.968 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 1232896, 2024-09-17T16:17:50.968 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 301, 2024-09-17T16:17:50.968 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-17T16:17:50.968 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 1505, 2024-09-17T16:17:50.968 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T16:17:50.968 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 1, 2024-09-17T16:17:50.968 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 301, 2024-09-17T16:17:50.968 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-17T16:17:50.968 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-09-17T16:17:50.968 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 301, 2024-09-17T16:17:50.968 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-17T16:17:50.969 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-17T16:17:50.969 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-17T16:17:50.969 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 301, 2024-09-17T16:17:50.969 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 1204, 2024-09-17T16:17:50.969 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-17T16:17:50.969 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T16:17:50.969 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-17T16:17:50.969 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-17T16:17:50.969 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-17T16:17:50.969 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-17T16:17:50.969 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-17T16:17:50.969 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T16:17:50.969 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T16:17:50.970 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-17T16:17:50.970 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-17T16:17:50.970 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-17T16:17:50.970 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-17T16:17:50.970 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-17T16:17:50.970 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-17T16:17:50.970 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-17T16:17:50.970 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-17T16:17:50.970 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-17T16:17:50.970 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-17T16:17:50.970 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-17T16:17:50.970 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-17T16:17:50.970 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-17T16:17:50.971 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-17T16:17:50.971 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-17T16:17:50.971 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-17T16:17:50.971 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.971 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-17T16:17:50.971 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:17:50.971 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:17:50.971 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:17:50.971 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:17:50.971 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-17T16:17:50.971 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.971 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-17T16:17:50.971 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:17:50.972 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:17:50.972 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:17:50.972 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:17:50.972 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-17T16:17:50.972 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.972 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-17T16:17:50.972 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-17T16:17:50.972 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-17T16:17:50.972 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-17T16:17:50.972 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-17T16:17:50.972 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.972 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-17T16:17:50.972 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.973 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-17T16:17:50.973 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 301 2024-09-17T16:17:50.973 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:17:50.973 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.973 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-17T16:17:50.973 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-17T16:17:50.973 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-17T16:17:50.973 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-17T16:17:50.973 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.973 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-17T16:17:50.973 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-17T16:17:50.973 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-17T16:17:50.973 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 64, 2024-09-17T16:17:50.974 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-17T16:17:50.974 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-17T16:17:50.974 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-17T16:17:50.974 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:17:50.974 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.974 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.974 INFO:tasks.workunit.client.0.smithi162.stdout: "peer": "4(2)", 2024-09-17T16:17:50.974 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s2", 2024-09-17T16:17:50.974 INFO:tasks.workunit.client.0.smithi162.stdout: "shared": "2", 2024-09-17T16:17:50.974 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "54'301", 2024-09-17T16:17:50.974 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "54'301", 2024-09-17T16:17:50.974 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "54'200", 2024-09-17T16:17:50.974 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 301, 2024-09-17T16:17:50.975 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-17T16:17:50.975 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-17T16:17:50.975 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-17T16:17:50.975 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 46, 2024-09-17T16:17:50.975 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 46, 2024-09-17T16:17:50.975 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 64, 2024-09-17T16:17:50.975 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 62, 2024-09-17T16:17:50.975 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:17:50.975 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 46, 2024-09-17T16:17:50.975 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-17T16:17:50.975 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-17T16:17:50.975 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 62, 2024-09-17T16:17:50.975 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 62, 2024-09-17T16:17:50.976 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 46, 2024-09-17T16:17:50.976 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:17:50.976 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.976 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:17:50.976 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.976 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.976 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-17T16:17:50.976 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.976 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-17T16:17:50.976 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "54'301", 2024-09-17T16:17:50.976 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 335, 2024-09-17T16:17:50.976 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 54, 2024-09-17T16:17:50.976 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+undersized+degraded", 2024-09-17T16:17:50.976 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-17T16:17:09.061093+0000", 2024-09-17T16:17:50.977 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-17T16:15:54.716937+0000", 2024-09-17T16:17:50.977 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-17T16:17:09.061093+0000", 2024-09-17T16:17:50.977 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-17T16:17:09.061093+0000", 2024-09-17T16:17:50.977 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-17T16:15:52.464689+0000", 2024-09-17T16:17:50.977 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-17T16:15:54.716937+0000", 2024-09-17T16:17:50.977 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-17T16:15:54.716937+0000", 2024-09-17T16:17:50.977 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-17T16:17:09.061093+0000", 2024-09-17T16:17:50.977 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-17T16:15:54.375192+0000", 2024-09-17T16:17:50.977 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-17T16:15:54.374952+0000", 2024-09-17T16:17:50.977 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 62, 2024-09-17T16:17:50.977 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "54'200", 2024-09-17T16:17:50.977 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "54'200", 2024-09-17T16:17:50.977 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 46, 2024-09-17T16:17:50.977 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:17:50.978 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-17T16:17:50.978 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-17T16:17:50.978 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:17:50.978 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.978 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:17:50.978 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.978 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:15:45.736175+0000", 2024-09-17T16:17:50.978 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-17T16:17:50.978 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-17T16:17:50.978 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 200, 2024-09-17T16:17:50.978 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-17T16:17:50.978 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-17T16:17:50.978 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-17T16:17:50.978 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-17T16:17:50.979 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-17T16:17:50.979 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-17T16:17:50.979 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-17T16:17:50.979 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-17T16:17:50.979 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-17T16:17:50.979 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-17T16:17:50.979 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-18T23:50:50.279195+0000", 2024-09-17T16:17:50.979 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-17T16:17:50.979 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-17T16:17:50.979 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-17T16:17:50.979 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-17T16:17:50.979 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 1232896, 2024-09-17T16:17:50.979 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 301, 2024-09-17T16:17:50.980 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-17T16:17:50.980 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 1505, 2024-09-17T16:17:50.980 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T16:17:50.980 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 0, 2024-09-17T16:17:50.980 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 301, 2024-09-17T16:17:50.980 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-17T16:17:50.980 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-09-17T16:17:50.980 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 301, 2024-09-17T16:17:50.980 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-17T16:17:50.980 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-17T16:17:50.980 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-17T16:17:50.980 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 301, 2024-09-17T16:17:50.980 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 1204, 2024-09-17T16:17:50.980 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-17T16:17:50.981 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T16:17:50.981 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-17T16:17:50.981 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-17T16:17:50.981 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-17T16:17:50.981 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-17T16:17:50.981 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-17T16:17:50.981 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T16:17:50.981 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T16:17:50.981 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-17T16:17:50.981 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-17T16:17:50.981 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-17T16:17:50.981 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-17T16:17:50.981 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-17T16:17:50.982 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-17T16:17:50.982 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-17T16:17:50.982 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-17T16:17:50.982 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-17T16:17:50.982 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-17T16:17:50.982 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-17T16:17:50.982 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-17T16:17:50.982 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-17T16:17:50.982 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-17T16:17:50.982 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-17T16:17:50.982 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-17T16:17:50.982 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.982 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-17T16:17:50.983 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:17:50.983 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:17:50.983 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:17:50.983 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:17:50.983 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-17T16:17:50.983 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.983 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-17T16:17:50.983 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:17:50.983 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:17:50.983 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:17:50.983 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:17:50.983 INFO:tasks.workunit.client.0.smithi162.stdout: 2147483647 2024-09-17T16:17:50.983 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.983 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-17T16:17:50.984 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-17T16:17:50.984 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-17T16:17:50.984 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-17T16:17:50.984 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-17T16:17:50.984 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.984 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-17T16:17:50.984 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.984 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-17T16:17:50.984 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 301 2024-09-17T16:17:50.984 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:17:50.984 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.984 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-17T16:17:50.984 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-17T16:17:50.984 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-17T16:17:50.985 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-17T16:17:50.985 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.985 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-17T16:17:50.985 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-17T16:17:50.985 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-17T16:17:50.985 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 64, 2024-09-17T16:17:50.985 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-17T16:17:50.985 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-17T16:17:50.985 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-17T16:17:50.985 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:17:50.985 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:17:50.986 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.986 INFO:tasks.workunit.client.0.smithi162.stdout: "recovery_state": [ 2024-09-17T16:17:50.986 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.986 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "Started/Primary/Active", 2024-09-17T16:17:50.986 INFO:tasks.workunit.client.0.smithi162.stdout: "enter_time": "2024-09-17T16:17:46.147764+0000", 2024-09-17T16:17:50.986 INFO:tasks.workunit.client.0.smithi162.stdout: "might_have_unfound": [ 2024-09-17T16:17:50.986 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.986 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "0(3)", 2024-09-17T16:17:50.986 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "already probed" 2024-09-17T16:17:50.986 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.986 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.986 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "1(1)", 2024-09-17T16:17:50.986 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "already probed" 2024-09-17T16:17:50.986 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.987 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.987 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "2(4)", 2024-09-17T16:17:50.987 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "osd is down" 2024-09-17T16:17:50.987 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.987 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.987 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "4(2)", 2024-09-17T16:17:50.987 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "already probed" 2024-09-17T16:17:50.987 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:17:50.987 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.987 INFO:tasks.workunit.client.0.smithi162.stdout: "recovery_progress": { 2024-09-17T16:17:50.988 INFO:tasks.workunit.client.0.smithi162.stdout: "backfill_targets": [], 2024-09-17T16:17:50.988 INFO:tasks.workunit.client.0.smithi162.stdout: "waiting_on_backfill": [], 2024-09-17T16:17:50.988 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill_started": "MIN", 2024-09-17T16:17:50.988 INFO:tasks.workunit.client.0.smithi162.stdout: "backfill_info": { 2024-09-17T16:17:50.988 INFO:tasks.workunit.client.0.smithi162.stdout: "begin": "MIN", 2024-09-17T16:17:50.988 INFO:tasks.workunit.client.0.smithi162.stdout: "end": "MIN", 2024-09-17T16:17:50.988 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": [] 2024-09-17T16:17:50.988 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.988 INFO:tasks.workunit.client.0.smithi162.stdout: "peer_backfill_info": [], 2024-09-17T16:17:50.988 INFO:tasks.workunit.client.0.smithi162.stdout: "backfills_in_flight": [], 2024-09-17T16:17:50.989 INFO:tasks.workunit.client.0.smithi162.stdout: "recovering": [], 2024-09-17T16:17:50.989 INFO:tasks.workunit.client.0.smithi162.stdout: "pg_backend": { 2024-09-17T16:17:50.989 INFO:tasks.workunit.client.0.smithi162.stdout: "recovery_ops": [], 2024-09-17T16:17:50.989 INFO:tasks.workunit.client.0.smithi162.stdout: "read_ops": [] 2024-09-17T16:17:50.989 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:17:50.989 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:17:50.989 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.989 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:17:50.989 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "Started", 2024-09-17T16:17:50.989 INFO:tasks.workunit.client.0.smithi162.stdout: "enter_time": "2024-09-17T16:17:45.326540+0000" 2024-09-17T16:17:50.989 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:17:50.990 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:17:50.990 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubber": { 2024-09-17T16:17:50.990 INFO:tasks.workunit.client.0.smithi162.stdout: "active": false, 2024-09-17T16:17:50.990 INFO:tasks.workunit.client.0.smithi162.stdout: "must_scrub": false, 2024-09-17T16:17:50.990 INFO:tasks.workunit.client.0.smithi162.stdout: "must_deep_scrub": false, 2024-09-17T16:17:50.990 INFO:tasks.workunit.client.0.smithi162.stdout: "must_repair": false, 2024-09-17T16:17:50.990 INFO:tasks.workunit.client.0.smithi162.stdout: "need_auto": false, 2024-09-17T16:17:50.990 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_reg_stamp": "2024-09-18T23:50:50.279195+0000", 2024-09-17T16:17:50.990 INFO:tasks.workunit.client.0.smithi162.stdout: "schedule": "not registered for scrubbing" 2024-09-17T16:17:50.990 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:17:50.990 INFO:tasks.workunit.client.0.smithi162.stdout: "agent_state": {} 2024-09-17T16:17:50.990 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-09-17T16:17:50.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:590: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-17T16:17:50.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:590: TEST_ec_backfill_unfound: grep -q obj250 2024-09-17T16:17:51.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:592: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-17T16:17:51.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:592: TEST_ec_backfill_unfound: jq .available_might_have_unfound 2024-09-17T16:17:51.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:592: TEST_ec_backfill_unfound: check=true 2024-09-17T16:17:51.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: test true == true 2024-09-17T16:17:51.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-17T16:17:51.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].status' 2024-09-17T16:17:51.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: eval 'check="osd' is 'down"' 2024-09-17T16:17:51.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: check='osd is down' 2024-09-17T16:17:51.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:596: TEST_ec_backfill_unfound: test 'osd is down' == 'osd is down' 2024-09-17T16:17:51.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-17T16:17:51.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].osd' 2024-09-17T16:17:51.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: eval 'check="2(4)"' 2024-09-17T16:17:51.422 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: check='2(4)' 2024-09-17T16:17:51.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:599: TEST_ec_backfill_unfound: test '2(4)' == '2(4)' 2024-09-17T16:17:51.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:601: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-09-17T16:17:51.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:17:51.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:17:51.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T16:17:51.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:17:51.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:17:51.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:17:51.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:17:51.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:17:51.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:17:51.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:17:51.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:17:51.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:17:51.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:17:51.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:17:51.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:17:51.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:17:51.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:17:51.423 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:17:51.423 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:17:51.423 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:17:51.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:17:51.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:17:51.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:17:51.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:17:51.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:17:51.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:17:51.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:17:51.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:17:51.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:17:51.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:17:51.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:17:51.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:17:51.425 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T16:17:51.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T16:17:51.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:17:51.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-17T16:17:51.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T16:17:51.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:17:51.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:17:51.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:17:51.448 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:17:51.446+0000 7ff48bfd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:17:51.449 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:17:51.448+0000 7ff48bfd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:17:51.450 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:17:51.449+0000 7ff48bfd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:17:51.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T16:17:51.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:17:51.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:17:51.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:17:51.844 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:17:51.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:17:51.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:17:51.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:17:51.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:17:51.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:17:52.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:17:52.800 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:17:52.798+0000 7ff48bfd9640 -1 Falling back to public interface 2024-09-17T16:17:53.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:17:53.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:17:53.288 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:17:53.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:17:53.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:17:53.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:17:53.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:17:54.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:17:54.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:17:54.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:17:54.723 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:17:54.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:17:54.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:17:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:17:55.903 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:17:55.901+0000 7ff48bfd9640 -1 osd.2 61 log_to_monitors true 2024-09-17T16:17:56.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:17:56.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:17:56.137 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:17:56.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:17:56.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:17:56.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:17:56.629 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 66 up_thru 22 down_at 62 last_clean_interval [56,61) [v2:127.0.0.1:6818/3439558227,v1:127.0.0.1:6819/3439558227] [v2:127.0.0.1:6820/3439558227,v1:127.0.0.1:6821/3439558227] exists,up d09e1f7c-4713-45c7-9eb5-9f689f2aca77 2024-09-17T16:17:56.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:17:56.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:17:56.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:17:56.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:604: TEST_ec_backfill_unfound: timeout 5 rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-09-17T16:18:01.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:605: TEST_ec_backfill_unfound: test 124 = 124 2024-09-17T16:18:01.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:607: TEST_ec_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-17T16:18:01.924 INFO:tasks.workunit.client.0.smithi162.stderr:pg has no unfound objects 2024-09-17T16:18:01.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:609: TEST_ec_backfill_unfound: wait_for_clean 2024-09-17T16:18:01.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:18:01.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:18:01.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:18:01.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:18:01.936 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:18:01.936 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:18:01.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:18:01.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:18:01.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:18:02.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:18:02.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:18:02.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:18:02.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:18:02.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:18:02.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:18:02.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:18:02.415 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:18:02.415 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:18:02.416 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:18:02.416 INFO:tasks.workunit.client.0.smithi162.stderr:4' 2024-09-17T16:18:02.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:18:02.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:18:02.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:18:02.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803841 2024-09-17T16:18:02.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803841 2024-09-17T16:18:02.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841' 2024-09-17T16:18:02.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:18:02.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:18:02.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574910 2024-09-17T16:18:02.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574910 2024-09-17T16:18:02.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841 1-55834574910' 2024-09-17T16:18:02.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:18:02.744 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:18:02.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841540 2024-09-17T16:18:02.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841540 2024-09-17T16:18:02.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841 1-55834574910 2-283467841540' 2024-09-17T16:18:02.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:18:02.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:18:02.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018923 2024-09-17T16:18:02.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018923 2024-09-17T16:18:02.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841 1-55834574910 2-283467841540 3-128849018923' 2024-09-17T16:18:02.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:18:02.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:18:03.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659171 2024-09-17T16:18:03.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659171 2024-09-17T16:18:03.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841 1-55834574910 2-283467841540 3-128849018923 4-176093659171' 2024-09-17T16:18:03.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:18:03.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803841 2024-09-17T16:18:03.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:18:03.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:18:03.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803841 2024-09-17T16:18:03.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:18:03.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803841 2024-09-17T16:18:03.090 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803841 2024-09-17T16:18:03.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803841' 2024-09-17T16:18:03.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:18:03.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803841 -lt 25769803841 2024-09-17T16:18:03.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:18:03.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574910 2024-09-17T16:18:03.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:18:03.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:18:03.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574910 2024-09-17T16:18:03.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:18:03.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574910 2024-09-17T16:18:03.479 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574910 2024-09-17T16:18:03.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574910' 2024-09-17T16:18:03.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:18:03.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574908 -lt 55834574910 2024-09-17T16:18:03.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:18:04.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:18:04.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:18:05.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574910 -lt 55834574910 2024-09-17T16:18:05.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:18:05.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-283467841540 2024-09-17T16:18:05.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:18:05.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:18:05.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-283467841540 2024-09-17T16:18:05.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:18:05.396 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 283467841540 2024-09-17T16:18:05.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841540 2024-09-17T16:18:05.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 283467841540' 2024-09-17T16:18:05.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:18:05.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841540 -lt 283467841540 2024-09-17T16:18:05.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:18:05.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018923 2024-09-17T16:18:05.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:18:05.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:18:05.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018923 2024-09-17T16:18:05.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:18:05.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018923 2024-09-17T16:18:05.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018923' 2024-09-17T16:18:05.900 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 128849018923 2024-09-17T16:18:05.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:18:06.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018923 -lt 128849018923 2024-09-17T16:18:06.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:18:06.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-176093659171 2024-09-17T16:18:06.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:18:06.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:18:06.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-176093659171 2024-09-17T16:18:06.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:18:06.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659171 2024-09-17T16:18:06.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 176093659171' 2024-09-17T16:18:06.394 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 176093659171 2024-09-17T16:18:06.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:18:06.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659172 -lt 176093659171 2024-09-17T16:18:06.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:18:06.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:18:06.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:18:07.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:18:07.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:18:07.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:18:07.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:18:07.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:18:07.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:18:07.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:18:07.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:18:07.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T16:18:07.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:18:07.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:18:07.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:18:08.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T16:18:08.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T16:18:08.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T16:18:08.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T16:18:08.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T16:18:08.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T16:18:08.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T16:18:08.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:18:08.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:18:08.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:18:08.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:18:08.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:18:08.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:18:08.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:18:08.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:18:08.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:18:08.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:18:08.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:18:09.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:18:09.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:18:09.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:18:09.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: seq 1 300 2024-09-17T16:18:09.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj1 = obj250 ']' 2024-09-17T16:18:09.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj2 = obj250 ']' 2024-09-17T16:18:09.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj3 = obj250 ']' 2024-09-17T16:18:09.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj4 = obj250 ']' 2024-09-17T16:18:09.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj5 = obj250 ']' 2024-09-17T16:18:09.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj6 = obj250 ']' 2024-09-17T16:18:09.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj7 = obj250 ']' 2024-09-17T16:18:09.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj8 = obj250 ']' 2024-09-17T16:18:09.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj9 = obj250 ']' 2024-09-17T16:18:09.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj10 = obj250 ']' 2024-09-17T16:18:09.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj11 = obj250 ']' 2024-09-17T16:18:09.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj12 = obj250 ']' 2024-09-17T16:18:09.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj13 = obj250 ']' 2024-09-17T16:18:09.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj14 = obj250 ']' 2024-09-17T16:18:09.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj15 = obj250 ']' 2024-09-17T16:18:09.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj16 = obj250 ']' 2024-09-17T16:18:09.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj17 = obj250 ']' 2024-09-17T16:18:09.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj18 = obj250 ']' 2024-09-17T16:18:09.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj19 = obj250 ']' 2024-09-17T16:18:09.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj20 = obj250 ']' 2024-09-17T16:18:09.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-09-17T16:18:09.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:09.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:09.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj21 = obj250 ']' 2024-09-17T16:18:09.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj22 = obj250 ']' 2024-09-17T16:18:10.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj23 = obj250 ']' 2024-09-17T16:18:10.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj24 = obj250 ']' 2024-09-17T16:18:10.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj25 = obj250 ']' 2024-09-17T16:18:10.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj26 = obj250 ']' 2024-09-17T16:18:10.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj27 = obj250 ']' 2024-09-17T16:18:10.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj28 = obj250 ']' 2024-09-17T16:18:10.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj29 = obj250 ']' 2024-09-17T16:18:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj30 = obj250 ']' 2024-09-17T16:18:10.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj31 = obj250 ']' 2024-09-17T16:18:10.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj32 = obj250 ']' 2024-09-17T16:18:10.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj33 = obj250 ']' 2024-09-17T16:18:10.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj34 = obj250 ']' 2024-09-17T16:18:10.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj35 = obj250 ']' 2024-09-17T16:18:10.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj36 = obj250 ']' 2024-09-17T16:18:10.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj37 = obj250 ']' 2024-09-17T16:18:10.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj38 = obj250 ']' 2024-09-17T16:18:10.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj39 = obj250 ']' 2024-09-17T16:18:10.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj40 = obj250 ']' 2024-09-17T16:18:10.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj41 = obj250 ']' 2024-09-17T16:18:10.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj42 = obj250 ']' 2024-09-17T16:18:10.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj43 = obj250 ']' 2024-09-17T16:18:10.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj44 = obj250 ']' 2024-09-17T16:18:10.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj45 = obj250 ']' 2024-09-17T16:18:10.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj46 = obj250 ']' 2024-09-17T16:18:10.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj47 = obj250 ']' 2024-09-17T16:18:10.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj48 = obj250 ']' 2024-09-17T16:18:10.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj49 = obj250 ']' 2024-09-17T16:18:10.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj50 = obj250 ']' 2024-09-17T16:18:10.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj51 = obj250 ']' 2024-09-17T16:18:10.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-09-17T16:18:10.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:10.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:10.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj52 = obj250 ']' 2024-09-17T16:18:10.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj53 = obj250 ']' 2024-09-17T16:18:11.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj54 = obj250 ']' 2024-09-17T16:18:11.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj55 = obj250 ']' 2024-09-17T16:18:11.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj56 = obj250 ']' 2024-09-17T16:18:11.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj57 = obj250 ']' 2024-09-17T16:18:11.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj58 = obj250 ']' 2024-09-17T16:18:11.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj59 = obj250 ']' 2024-09-17T16:18:11.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj60 = obj250 ']' 2024-09-17T16:18:11.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj61 = obj250 ']' 2024-09-17T16:18:11.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj62 = obj250 ']' 2024-09-17T16:18:11.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj63 = obj250 ']' 2024-09-17T16:18:11.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj64 = obj250 ']' 2024-09-17T16:18:11.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj65 = obj250 ']' 2024-09-17T16:18:11.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj66 = obj250 ']' 2024-09-17T16:18:11.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj67 = obj250 ']' 2024-09-17T16:18:11.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj68 = obj250 ']' 2024-09-17T16:18:11.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj69 = obj250 ']' 2024-09-17T16:18:11.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj70 = obj250 ']' 2024-09-17T16:18:11.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj71 = obj250 ']' 2024-09-17T16:18:11.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj72 = obj250 ']' 2024-09-17T16:18:11.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj73 = obj250 ']' 2024-09-17T16:18:11.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj74 = obj250 ']' 2024-09-17T16:18:11.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj75 = obj250 ']' 2024-09-17T16:18:11.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj76 = obj250 ']' 2024-09-17T16:18:11.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj77 = obj250 ']' 2024-09-17T16:18:11.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj78 = obj250 ']' 2024-09-17T16:18:11.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj79 = obj250 ']' 2024-09-17T16:18:11.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj80 = obj250 ']' 2024-09-17T16:18:11.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj81 = obj250 ']' 2024-09-17T16:18:11.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-09-17T16:18:11.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:11.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:11.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj82 = obj250 ']' 2024-09-17T16:18:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj83 = obj250 ']' 2024-09-17T16:18:12.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj84 = obj250 ']' 2024-09-17T16:18:12.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj85 = obj250 ']' 2024-09-17T16:18:12.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj86 = obj250 ']' 2024-09-17T16:18:12.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj87 = obj250 ']' 2024-09-17T16:18:12.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj88 = obj250 ']' 2024-09-17T16:18:12.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj89 = obj250 ']' 2024-09-17T16:18:12.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj90 = obj250 ']' 2024-09-17T16:18:12.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj91 = obj250 ']' 2024-09-17T16:18:12.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj92 = obj250 ']' 2024-09-17T16:18:12.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj93 = obj250 ']' 2024-09-17T16:18:12.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj94 = obj250 ']' 2024-09-17T16:18:12.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj95 = obj250 ']' 2024-09-17T16:18:12.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj96 = obj250 ']' 2024-09-17T16:18:12.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj97 = obj250 ']' 2024-09-17T16:18:12.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj98 = obj250 ']' 2024-09-17T16:18:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj99 = obj250 ']' 2024-09-17T16:18:12.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj100 = obj250 ']' 2024-09-17T16:18:12.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj101 = obj250 ']' 2024-09-17T16:18:12.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj101 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj102 = obj250 ']' 2024-09-17T16:18:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj102 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj103 = obj250 ']' 2024-09-17T16:18:12.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj103 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj104 = obj250 ']' 2024-09-17T16:18:12.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj104 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj105 = obj250 ']' 2024-09-17T16:18:12.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj105 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj106 = obj250 ']' 2024-09-17T16:18:12.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj106 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj107 = obj250 ']' 2024-09-17T16:18:12.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj107 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj108 = obj250 ']' 2024-09-17T16:18:12.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj108 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj109 = obj250 ']' 2024-09-17T16:18:12.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj109 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj110 = obj250 ']' 2024-09-17T16:18:12.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj110 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj111 = obj250 ']' 2024-09-17T16:18:12.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj111 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj112 = obj250 ']' 2024-09-17T16:18:12.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj112 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj113 = obj250 ']' 2024-09-17T16:18:12.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj113 td/test-erasure-eio/CHECK 2024-09-17T16:18:12.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:12.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:12.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj114 = obj250 ']' 2024-09-17T16:18:12.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj114 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj115 = obj250 ']' 2024-09-17T16:18:13.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj115 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj116 = obj250 ']' 2024-09-17T16:18:13.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj116 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj117 = obj250 ']' 2024-09-17T16:18:13.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj117 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj118 = obj250 ']' 2024-09-17T16:18:13.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj118 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj119 = obj250 ']' 2024-09-17T16:18:13.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj119 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj120 = obj250 ']' 2024-09-17T16:18:13.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj120 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj121 = obj250 ']' 2024-09-17T16:18:13.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj121 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj122 = obj250 ']' 2024-09-17T16:18:13.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj122 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj123 = obj250 ']' 2024-09-17T16:18:13.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj123 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj124 = obj250 ']' 2024-09-17T16:18:13.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj124 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj125 = obj250 ']' 2024-09-17T16:18:13.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj125 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj126 = obj250 ']' 2024-09-17T16:18:13.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj126 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj127 = obj250 ']' 2024-09-17T16:18:13.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj127 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj128 = obj250 ']' 2024-09-17T16:18:13.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj128 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj129 = obj250 ']' 2024-09-17T16:18:13.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj129 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj130 = obj250 ']' 2024-09-17T16:18:13.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj130 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj131 = obj250 ']' 2024-09-17T16:18:13.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj131 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj132 = obj250 ']' 2024-09-17T16:18:13.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj132 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj133 = obj250 ']' 2024-09-17T16:18:13.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj133 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj134 = obj250 ']' 2024-09-17T16:18:13.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj134 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj135 = obj250 ']' 2024-09-17T16:18:13.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj135 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj136 = obj250 ']' 2024-09-17T16:18:13.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj136 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj137 = obj250 ']' 2024-09-17T16:18:13.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj137 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj138 = obj250 ']' 2024-09-17T16:18:13.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj138 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj139 = obj250 ']' 2024-09-17T16:18:13.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj139 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj140 = obj250 ']' 2024-09-17T16:18:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj140 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj141 = obj250 ']' 2024-09-17T16:18:13.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj141 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj142 = obj250 ']' 2024-09-17T16:18:13.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj142 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj143 = obj250 ']' 2024-09-17T16:18:13.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj143 td/test-erasure-eio/CHECK 2024-09-17T16:18:13.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:13.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:13.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj144 = obj250 ']' 2024-09-17T16:18:13.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj144 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj145 = obj250 ']' 2024-09-17T16:18:14.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj145 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj146 = obj250 ']' 2024-09-17T16:18:14.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj146 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj147 = obj250 ']' 2024-09-17T16:18:14.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj147 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj148 = obj250 ']' 2024-09-17T16:18:14.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj148 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj149 = obj250 ']' 2024-09-17T16:18:14.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj149 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj150 = obj250 ']' 2024-09-17T16:18:14.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj150 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj151 = obj250 ']' 2024-09-17T16:18:14.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj151 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj152 = obj250 ']' 2024-09-17T16:18:14.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj152 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj153 = obj250 ']' 2024-09-17T16:18:14.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj153 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj154 = obj250 ']' 2024-09-17T16:18:14.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj154 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj155 = obj250 ']' 2024-09-17T16:18:14.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj155 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj156 = obj250 ']' 2024-09-17T16:18:14.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj156 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj157 = obj250 ']' 2024-09-17T16:18:14.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj157 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj158 = obj250 ']' 2024-09-17T16:18:14.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj158 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj159 = obj250 ']' 2024-09-17T16:18:14.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj159 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj160 = obj250 ']' 2024-09-17T16:18:14.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj160 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj161 = obj250 ']' 2024-09-17T16:18:14.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj161 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj162 = obj250 ']' 2024-09-17T16:18:14.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj162 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj163 = obj250 ']' 2024-09-17T16:18:14.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj163 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj164 = obj250 ']' 2024-09-17T16:18:14.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj164 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj165 = obj250 ']' 2024-09-17T16:18:14.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj165 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj166 = obj250 ']' 2024-09-17T16:18:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj166 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj167 = obj250 ']' 2024-09-17T16:18:14.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj167 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj168 = obj250 ']' 2024-09-17T16:18:14.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj168 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj169 = obj250 ']' 2024-09-17T16:18:14.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj169 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj170 = obj250 ']' 2024-09-17T16:18:14.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj170 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj171 = obj250 ']' 2024-09-17T16:18:14.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj171 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj172 = obj250 ']' 2024-09-17T16:18:14.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj172 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj173 = obj250 ']' 2024-09-17T16:18:14.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj173 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj174 = obj250 ']' 2024-09-17T16:18:14.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj174 td/test-erasure-eio/CHECK 2024-09-17T16:18:14.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:14.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:14.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj175 = obj250 ']' 2024-09-17T16:18:14.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj175 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj176 = obj250 ']' 2024-09-17T16:18:15.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj176 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj177 = obj250 ']' 2024-09-17T16:18:15.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj177 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj178 = obj250 ']' 2024-09-17T16:18:15.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj178 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj179 = obj250 ']' 2024-09-17T16:18:15.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj179 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj180 = obj250 ']' 2024-09-17T16:18:15.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj180 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj181 = obj250 ']' 2024-09-17T16:18:15.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj181 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj182 = obj250 ']' 2024-09-17T16:18:15.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj182 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj183 = obj250 ']' 2024-09-17T16:18:15.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj183 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj184 = obj250 ']' 2024-09-17T16:18:15.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj184 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj185 = obj250 ']' 2024-09-17T16:18:15.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj185 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj186 = obj250 ']' 2024-09-17T16:18:15.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj186 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj187 = obj250 ']' 2024-09-17T16:18:15.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj187 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj188 = obj250 ']' 2024-09-17T16:18:15.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj188 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj189 = obj250 ']' 2024-09-17T16:18:15.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj189 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj190 = obj250 ']' 2024-09-17T16:18:15.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj190 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj191 = obj250 ']' 2024-09-17T16:18:15.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj191 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj192 = obj250 ']' 2024-09-17T16:18:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj192 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj193 = obj250 ']' 2024-09-17T16:18:15.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj193 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj194 = obj250 ']' 2024-09-17T16:18:15.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj194 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj195 = obj250 ']' 2024-09-17T16:18:15.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj195 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj196 = obj250 ']' 2024-09-17T16:18:15.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj196 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj197 = obj250 ']' 2024-09-17T16:18:15.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj197 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj198 = obj250 ']' 2024-09-17T16:18:15.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj198 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj199 = obj250 ']' 2024-09-17T16:18:15.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj199 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj200 = obj250 ']' 2024-09-17T16:18:15.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj200 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj201 = obj250 ']' 2024-09-17T16:18:15.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj201 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj202 = obj250 ']' 2024-09-17T16:18:15.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj202 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj203 = obj250 ']' 2024-09-17T16:18:15.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj203 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj204 = obj250 ']' 2024-09-17T16:18:15.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj204 td/test-erasure-eio/CHECK 2024-09-17T16:18:15.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:15.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:15.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj205 = obj250 ']' 2024-09-17T16:18:15.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj205 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj206 = obj250 ']' 2024-09-17T16:18:16.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj206 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj207 = obj250 ']' 2024-09-17T16:18:16.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj207 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj208 = obj250 ']' 2024-09-17T16:18:16.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj208 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj209 = obj250 ']' 2024-09-17T16:18:16.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj209 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj210 = obj250 ']' 2024-09-17T16:18:16.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj210 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj211 = obj250 ']' 2024-09-17T16:18:16.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj211 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj212 = obj250 ']' 2024-09-17T16:18:16.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj212 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj213 = obj250 ']' 2024-09-17T16:18:16.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj213 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj214 = obj250 ']' 2024-09-17T16:18:16.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj214 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj215 = obj250 ']' 2024-09-17T16:18:16.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj215 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj216 = obj250 ']' 2024-09-17T16:18:16.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj216 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj217 = obj250 ']' 2024-09-17T16:18:16.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj217 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj218 = obj250 ']' 2024-09-17T16:18:16.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj218 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj219 = obj250 ']' 2024-09-17T16:18:16.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj219 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj220 = obj250 ']' 2024-09-17T16:18:16.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj220 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj221 = obj250 ']' 2024-09-17T16:18:16.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj221 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj222 = obj250 ']' 2024-09-17T16:18:16.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj222 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj223 = obj250 ']' 2024-09-17T16:18:16.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj223 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj224 = obj250 ']' 2024-09-17T16:18:16.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj224 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj225 = obj250 ']' 2024-09-17T16:18:16.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj225 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj226 = obj250 ']' 2024-09-17T16:18:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj226 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj227 = obj250 ']' 2024-09-17T16:18:16.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj227 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj228 = obj250 ']' 2024-09-17T16:18:16.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj228 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj229 = obj250 ']' 2024-09-17T16:18:16.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj229 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj230 = obj250 ']' 2024-09-17T16:18:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj230 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj231 = obj250 ']' 2024-09-17T16:18:16.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj231 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj232 = obj250 ']' 2024-09-17T16:18:16.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj232 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj233 = obj250 ']' 2024-09-17T16:18:16.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj233 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj234 = obj250 ']' 2024-09-17T16:18:16.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj234 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj235 = obj250 ']' 2024-09-17T16:18:16.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj235 td/test-erasure-eio/CHECK 2024-09-17T16:18:16.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:16.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:16.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj236 = obj250 ']' 2024-09-17T16:18:16.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj236 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj237 = obj250 ']' 2024-09-17T16:18:17.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj237 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj238 = obj250 ']' 2024-09-17T16:18:17.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj238 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj239 = obj250 ']' 2024-09-17T16:18:17.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj239 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj240 = obj250 ']' 2024-09-17T16:18:17.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj240 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj241 = obj250 ']' 2024-09-17T16:18:17.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj241 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj242 = obj250 ']' 2024-09-17T16:18:17.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj242 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj243 = obj250 ']' 2024-09-17T16:18:17.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj243 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj244 = obj250 ']' 2024-09-17T16:18:17.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj244 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj245 = obj250 ']' 2024-09-17T16:18:17.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj245 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj246 = obj250 ']' 2024-09-17T16:18:17.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj246 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj247 = obj250 ']' 2024-09-17T16:18:17.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj247 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj248 = obj250 ']' 2024-09-17T16:18:17.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj248 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj249 = obj250 ']' 2024-09-17T16:18:17.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj249 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj250 = obj250 ']' 2024-09-17T16:18:17.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:615: TEST_ec_backfill_unfound: rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.454 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj250: (2) No such file or directory 2024-09-17T16:18:17.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj251 = obj250 ']' 2024-09-17T16:18:17.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj251 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj252 = obj250 ']' 2024-09-17T16:18:17.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj252 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj253 = obj250 ']' 2024-09-17T16:18:17.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj253 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj254 = obj250 ']' 2024-09-17T16:18:17.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj254 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj255 = obj250 ']' 2024-09-17T16:18:17.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj255 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj256 = obj250 ']' 2024-09-17T16:18:17.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj256 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj257 = obj250 ']' 2024-09-17T16:18:17.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj257 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj258 = obj250 ']' 2024-09-17T16:18:17.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj258 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj259 = obj250 ']' 2024-09-17T16:18:17.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj259 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj260 = obj250 ']' 2024-09-17T16:18:17.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj260 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj261 = obj250 ']' 2024-09-17T16:18:17.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj261 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj262 = obj250 ']' 2024-09-17T16:18:17.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj262 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj263 = obj250 ']' 2024-09-17T16:18:17.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj263 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj264 = obj250 ']' 2024-09-17T16:18:17.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj264 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj265 = obj250 ']' 2024-09-17T16:18:17.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj265 td/test-erasure-eio/CHECK 2024-09-17T16:18:17.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:17.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:17.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj266 = obj250 ']' 2024-09-17T16:18:17.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj266 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj267 = obj250 ']' 2024-09-17T16:18:18.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj267 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj268 = obj250 ']' 2024-09-17T16:18:18.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj268 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj269 = obj250 ']' 2024-09-17T16:18:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj269 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj270 = obj250 ']' 2024-09-17T16:18:18.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj270 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj271 = obj250 ']' 2024-09-17T16:18:18.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj271 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj272 = obj250 ']' 2024-09-17T16:18:18.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj272 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj273 = obj250 ']' 2024-09-17T16:18:18.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj273 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj274 = obj250 ']' 2024-09-17T16:18:18.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj274 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj275 = obj250 ']' 2024-09-17T16:18:18.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj275 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj276 = obj250 ']' 2024-09-17T16:18:18.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj276 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj277 = obj250 ']' 2024-09-17T16:18:18.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj277 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj278 = obj250 ']' 2024-09-17T16:18:18.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj278 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj279 = obj250 ']' 2024-09-17T16:18:18.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj279 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj280 = obj250 ']' 2024-09-17T16:18:18.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj280 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj281 = obj250 ']' 2024-09-17T16:18:18.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj281 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj282 = obj250 ']' 2024-09-17T16:18:18.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj282 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj283 = obj250 ']' 2024-09-17T16:18:18.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj283 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj284 = obj250 ']' 2024-09-17T16:18:18.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj284 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj285 = obj250 ']' 2024-09-17T16:18:18.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj285 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj286 = obj250 ']' 2024-09-17T16:18:18.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj286 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj287 = obj250 ']' 2024-09-17T16:18:18.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj287 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj288 = obj250 ']' 2024-09-17T16:18:18.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj288 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj289 = obj250 ']' 2024-09-17T16:18:18.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj289 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj290 = obj250 ']' 2024-09-17T16:18:18.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj290 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj291 = obj250 ']' 2024-09-17T16:18:18.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj291 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj292 = obj250 ']' 2024-09-17T16:18:18.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj292 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj293 = obj250 ']' 2024-09-17T16:18:18.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj293 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj294 = obj250 ']' 2024-09-17T16:18:18.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj294 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj295 = obj250 ']' 2024-09-17T16:18:18.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj295 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj296 = obj250 ']' 2024-09-17T16:18:18.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj296 td/test-erasure-eio/CHECK 2024-09-17T16:18:18.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:18.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:18.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj297 = obj250 ']' 2024-09-17T16:18:18.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj297 td/test-erasure-eio/CHECK 2024-09-17T16:18:19.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:19.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:19.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj298 = obj250 ']' 2024-09-17T16:18:19.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj298 td/test-erasure-eio/CHECK 2024-09-17T16:18:19.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:19.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:19.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj299 = obj250 ']' 2024-09-17T16:18:19.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj299 td/test-erasure-eio/CHECK 2024-09-17T16:18:19.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:19.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:18:19.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj300 = obj250 ']' 2024-09-17T16:18:19.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj300 td/test-erasure-eio/CHECK 2024-09-17T16:18:19.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:19.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:622: TEST_ec_backfill_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:18:19.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:624: TEST_ec_backfill_unfound: delete_erasure_coded_pool pool-jerasure 2024-09-17T16:18:19.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:18:19.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-17T16:18:19.642 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-17T16:18:19.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-17T16:18:20.252 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-17T16:18:20.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-17T16:18:20.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:18:20.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:18:20.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:18:20.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:18:20.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:18:20.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:18:20.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:18:20.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:18:20.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:18:20.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:18:20.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:18:20.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:18:20.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:18:20.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:18:20.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:18:20.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:18:20.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:18:20.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:18:20.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:18:20.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:18:20.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:18:20.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:18:20.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:18:20.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:18:20.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:18:20.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:18:20.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:18:20.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:18:20.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-17T16:18:20.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-17T16:18:20.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-17T16:18:20.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-17T16:18:20.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:18:20.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:18:20.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:18:20.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:18:20.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:18:20.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:18:20.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:18:20.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:18:20.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:18:20.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:18:20.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:18:20.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:18:20.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:18:20.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:18:20.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:18:20.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:18:20.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:18:20.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:18:20.440 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:18:20.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:18:20.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:18:20.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:18:20.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:18:20.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:18:20.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:18:20.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:18:20.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:18:20.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:18:20.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-17T16:18:20.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:18:20.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:18:20.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:18:20.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.87940 2024-09-17T16:18:20.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:18:20.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T16:18:20.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:18:20.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-17T16:18:20.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-17T16:18:20.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-17T16:18:20.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:18:20.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:18:20.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:18:20.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-17T16:18:20.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-17T16:18:20.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:18:20.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:18:20.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:18:20.912 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:18:20.912 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:18:20.912 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:18:20.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:18:20.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T16:18:21.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:18:21.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:18:21.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:18:21.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:18:21.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:18:21.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:18:21.122 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:18:21.122 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:18:21.122 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:18:21.123 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:18:21.123 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:18:21.123 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:18:21.123 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:18:21.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:18:21.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get fsid 2024-09-17T16:18:21.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:18:21.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:18:21.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:18:21.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:18:21.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:18:21.203 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:18:21.203 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:18:21.203 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:18:21.203 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:18:21.203 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:18:21.203 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:18:21.203 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:18:21.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:18:21.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get mon_host 2024-09-17T16:18:21.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-17T16:18:21.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-17T16:18:21.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:18:21.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:18:21.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:18:21.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-17T16:18:21.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:18:22.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:18:22.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:18:22.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:18:22.128 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:18:22.128 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:18:22.128 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:18:22.128 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:18:22.128 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:18:22.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:18:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-17T16:18:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T16:18:22.422 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T16:18:22.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:18:23.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-17T16:18:23.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:18:23.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:18:23.434 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:18:23.435 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:18:23.435 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:18:23.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:18:23.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-17T16:18:23.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-mon.a.asok log flush 2024-09-17T16:18:23.506 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:18:23.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-17T16:18:23.516 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:18:20.943+0000 7ff558a1cc80 0 load: jerasure load: lrc 2024-09-17T16:18:23.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_object_attr_read_error td/test-erasure-eio 2024-09-17T16:18:23.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:379: TEST_ec_object_attr_read_error: local dir=td/test-erasure-eio 2024-09-17T16:18:23.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:380: TEST_ec_object_attr_read_error: local objname=myobject 2024-09-17T16:18:23.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:382: TEST_ec_object_attr_read_error: setup_osds 7 2024-09-17T16:18:23.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-17T16:18:23.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-17T16:18:23.517 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-17T16:18:23.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-17T16:18:23.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:18:23.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-17T16:18:23.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:18:23.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:18:23.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:18:23.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:18:23.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-17T16:18:23.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:18:23.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:18:23.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:18:23.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:18:23.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-17T16:18:23.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-17T16:18:23.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:18:23.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:18:23.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:18:23.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:18:23.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:18:23.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:18:23.527 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:18:23.527 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:18:23.527 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:18:23.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:18:23.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:18:23.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:18:23.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:18:23.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:18:23.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:18:23.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:18:23.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:18:23.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:18:23.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:18:23.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:18:23.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:18:23.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:18:23.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-17T16:18:23.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:18:23.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1f5ad21a-36c5-49f4-bb17-5ba0e3a2995c 2024-09-17T16:18:23.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1f5ad21a-36c5-49f4-bb17-5ba0e3a2995c' 2024-09-17T16:18:23.529 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 1f5ad21a-36c5-49f4-bb17-5ba0e3a2995c 2024-09-17T16:18:23.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:18:23.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDPq+lmShglIBAA8PyPrOPjmAbmk1w2lL9Opg== 2024-09-17T16:18:23.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDPq+lmShglIBAA8PyPrOPjmAbmk1w2lL9Opg=="}' 2024-09-17T16:18:23.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1f5ad21a-36c5-49f4-bb17-5ba0e3a2995c -i td/test-erasure-eio/0/new.json 2024-09-17T16:18:23.789 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:18:23.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-17T16:18:23.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPq+lmShglIBAA8PyPrOPjmAbmk1w2lL9Opg== --osd-uuid 1f5ad21a-36c5-49f4-bb17-5ba0e3a2995c 2024-09-17T16:18:23.834 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:18:23.832+0000 7fd176e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:18:23.858 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:18:23.856+0000 7fd176e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:18:23.859 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:18:23.857+0000 7fd176e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:18:23.860 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:18:23.858+0000 7fd176e18640 -1 bdev(0x5644b9bf7c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:18:23.860 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:18:23.858+0000 7fd176e18640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-17T16:18:29.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-17T16:18:29.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:18:29.795 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:18:29.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:18:29.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:18:30.344 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:18:30.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:18:30.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:18:30.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:18:30.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:18:30.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:18:30.371 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:18:30.369+0000 7f4f3a032640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:18:30.377 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:18:30.375+0000 7f4f3a032640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:18:30.378 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:18:30.376+0000 7f4f3a032640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:18:30.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:18:30.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:18:30.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:18:30.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:18:30.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:18:30.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:30.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:18:30.784 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:18:30.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:30.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:31.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:31.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:18:31.729+0000 7f4f3a032640 -1 Falling back to public interface 2024-09-17T16:18:32.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:32.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:32.211 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:18:32.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:18:32.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:32.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:32.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:33.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:33.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:33.644 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:18:33.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:18:33.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:33.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:34.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:34.228 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:18:34.227+0000 7f4f3a032640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:18:35.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:35.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:35.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:18:35.033 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:18:35.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:35.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:35.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:36.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:36.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:36.543 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:18:36.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:18:36.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:36.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:37.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:38.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:38.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:38.034 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:18:38.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:18:38.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:38.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:38.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:39.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:39.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:39.453 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:18:39.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:18:39.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:39.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:39.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:40.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:40.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:40.885 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:18:40.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:18:40.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:40.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:41.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:42.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:42.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:42.382 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:18:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:18:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:42.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:42.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:43.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:43.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:43.929 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:18:43.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:18:43.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:43.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:44.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:45.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:45.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:45.383 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:18:45.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:18:45.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:45.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:45.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:46.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:46.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:46.973 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:18:46.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:18:46.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:46.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:47.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:48.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:48.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:48.733 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:18:48.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:18:48.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:48.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:49.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:50.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:50.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:50.653 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:18:50.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:18:50.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:50.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:51.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:52.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:52.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:52.733 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:18:52.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:18:52.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:52.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:53.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:54.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:54.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:54.281 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:18:54.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:18:54.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:54.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:54.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:55.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:55.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:55.749 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:18:55.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:18:55.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:55.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:56.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:18:57.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:18:57.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:18:57.628 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:18:57.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:18:57.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:18:57.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:18:58.069 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3766634130,v1:127.0.0.1:6803/3766634130] [v2:127.0.0.1:6804/3766634130,v1:127.0.0.1:6805/3766634130] exists,up 1f5ad21a-36c5-49f4-bb17-5ba0e3a2995c 2024-09-17T16:18:58.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:18:58.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:18:58.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:18:58.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:18:58.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-17T16:18:58.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:18:58.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:18:58.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T16:18:58.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:18:58.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:18:58.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:18:58.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:18:58.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:18:58.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:18:58.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:18:58.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:18:58.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:18:58.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:18:58.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:18:58.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:18:58.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:18:58.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:18:58.074 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:18:58.074 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:18:58.074 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:18:58.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:18:58.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:18:58.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:18:58.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:18:58.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:18:58.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:18:58.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:18:58.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:18:58.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:18:58.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:18:58.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:18:58.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:18:58.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:18:58.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:18:58.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:18:58.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=096a66fe-16ef-49dd-9fc9-0399f1621715 2024-09-17T16:18:58.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 096a66fe-16ef-49dd-9fc9-0399f1621715' 2024-09-17T16:18:58.076 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 096a66fe-16ef-49dd-9fc9-0399f1621715 2024-09-17T16:18:58.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:18:58.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDyq+lmmeVRBRAAENvwNfiFcGSVkDGVbVGT2Q== 2024-09-17T16:18:58.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDyq+lmmeVRBRAAENvwNfiFcGSVkDGVbVGT2Q=="}' 2024-09-17T16:18:58.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 096a66fe-16ef-49dd-9fc9-0399f1621715 -i td/test-erasure-eio/1/new.json 2024-09-17T16:18:58.575 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:18:58.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-17T16:18:58.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDyq+lmmeVRBRAAENvwNfiFcGSVkDGVbVGT2Q== --osd-uuid 096a66fe-16ef-49dd-9fc9-0399f1621715 2024-09-17T16:18:58.616 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:18:58.613+0000 7fa520ad6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:18:58.635 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:18:58.632+0000 7fa520ad6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:18:58.636 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:18:58.633+0000 7fa520ad6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:18:58.636 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:18:58.634+0000 7fa520ad6640 -1 bdev(0x55ff3404dc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-17T16:18:58.637 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:18:58.634+0000 7fa520ad6640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-17T16:19:04.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-17T16:19:04.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:19:04.545 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T16:19:04.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T16:19:04.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:19:05.080 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T16:19:05.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T16:19:05.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:19:05.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:19:05.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:19:05.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:19:05.107 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:19:05.104+0000 7fe27e214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:19:05.119 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:19:05.116+0000 7fe27e214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:19:05.120 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:19:05.117+0000 7fe27e214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:19:05.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T16:19:05.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:19:05.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:19:05.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:19:05.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:19:05.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:05.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:19:05.542 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:19:05.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:05.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:05.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:06.469 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:19:06.467+0000 7fe27e214640 -1 Falling back to public interface 2024-09-17T16:19:06.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:06.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:06.969 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:19:06.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:19:06.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:06.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:07.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:08.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:08.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:08.404 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:19:08.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:19:08.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:08.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:08.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:09.340 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:19:09.338+0000 7fe27e214640 -1 osd.1 0 log_to_monitors true 2024-09-17T16:19:09.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:09.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:09.796 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:19:09.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:19:09.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:09.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:10.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:11.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:11.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:11.191 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:19:11.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:19:11.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:11.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:11.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:12.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:12.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:12.585 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:19:12.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:19:12.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:12.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:13.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:14.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:14.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:14.003 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:19:14.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:19:14.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:14.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:14.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:15.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:15.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:15.512 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:19:15.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:19:15.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:15.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:15.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:16.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:16.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:16.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:19:16.916 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:19:16.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:16.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:17.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:18.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:18.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:18.366 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:19:18.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:19:18.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:18.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:18.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:19.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:19.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:19.893 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:19:19.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:19:19.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:19.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:20.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:21.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:21.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:21.314 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:19:21.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:19:21.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:21.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:21.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:22.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:22.737 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:19:22.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:22.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:19:22.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:22.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:23.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:24.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:24.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:24.484 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:19:24.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:19:24.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:24.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:25.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:26.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:26.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:26.587 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:19:26.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:19:26.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:26.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:27.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:28.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:28.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:28.437 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:19:28.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:19:28.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:28.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:29.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:30.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:30.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:30.235 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:19:30.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:19:30.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:30.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:31.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:32.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:32.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:32.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:19:32.301 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:19:32.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:32.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:32.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:33.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:33.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:33.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:19:33.838 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:19:33.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:33.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:19:34.243 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 14 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2209385797,v1:127.0.0.1:6811/2209385797] [v2:127.0.0.1:6812/2209385797,v1:127.0.0.1:6813/2209385797] exists,up 096a66fe-16ef-49dd-9fc9-0399f1621715 2024-09-17T16:19:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:19:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:19:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:19:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:19:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-17T16:19:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:19:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:19:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T16:19:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:19:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:19:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:19:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:19:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:19:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:19:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:19:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:19:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:19:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:19:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:19:34.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:19:34.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:19:34.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:19:34.249 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:19:34.249 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:19:34.249 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:19:34.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:19:34.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:19:34.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:19:34.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:19:34.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:19:34.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:19:34.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:19:34.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:19:34.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:19:34.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:19:34.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:19:34.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:19:34.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:19:34.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:19:34.250 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:19:34.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8cec7fae-c64c-4f3a-900b-b8e9b009991c 2024-09-17T16:19:34.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 8cec7fae-c64c-4f3a-900b-b8e9b009991c' 2024-09-17T16:19:34.250 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 8cec7fae-c64c-4f3a-900b-b8e9b009991c 2024-09-17T16:19:34.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:19:34.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAWrOlm6GOrDxAAn3ZfVTVZwZyIi7KgRhiPKQ== 2024-09-17T16:19:34.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAWrOlm6GOrDxAAn3ZfVTVZwZyIi7KgRhiPKQ=="}' 2024-09-17T16:19:34.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8cec7fae-c64c-4f3a-900b-b8e9b009991c -i td/test-erasure-eio/2/new.json 2024-09-17T16:19:34.710 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:19:34.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-17T16:19:34.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAWrOlm6GOrDxAAn3ZfVTVZwZyIi7KgRhiPKQ== --osd-uuid 8cec7fae-c64c-4f3a-900b-b8e9b009991c 2024-09-17T16:19:34.751 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:19:34.748+0000 7fdf6160c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:19:34.760 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:19:34.758+0000 7fdf6160c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:19:34.761 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:19:34.759+0000 7fdf6160c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:19:34.762 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:19:34.760+0000 7fdf6160c640 -1 bdev(0x5570f0785c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-17T16:19:34.762 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:19:34.760+0000 7fdf6160c640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-17T16:19:40.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-17T16:19:40.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:19:40.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T16:19:40.468 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T16:19:40.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:19:41.000 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T16:19:41.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T16:19:41.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:19:41.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:19:41.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:19:41.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:19:41.027 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:19:41.025+0000 7f2db65cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:19:41.039 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:19:41.037+0000 7f2db65cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:19:41.040 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:19:41.038+0000 7f2db65cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:19:41.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T16:19:41.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:19:41.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:19:41.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:19:41.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:19:41.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:41.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:19:41.454 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:19:41.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:41.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:19:41.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:42.632 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:19:42.630+0000 7f2db65cc640 -1 Falling back to public interface 2024-09-17T16:19:42.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:42.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:42.880 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:19:42.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:19:42.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:42.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:19:43.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:44.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:44.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:44.277 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:19:44.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:19:44.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:44.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:19:44.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:45.343 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:19:45.340+0000 7f2db65cc640 -1 osd.2 0 log_to_monitors true 2024-09-17T16:19:45.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:45.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:45.751 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:19:45.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:19:45.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:45.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:19:46.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:47.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:47.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:47.144 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:19:47.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:19:47.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:47.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:19:47.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:48.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:48.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:48.561 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:19:48.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:19:48.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:48.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:19:48.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:49.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:49.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:49.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:19:49.960 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:19:49.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:49.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:19:50.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:51.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:51.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:51.452 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:19:51.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:19:51.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:51.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:19:51.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:52.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:52.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:52.873 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:19:52.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:19:52.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:52.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:19:53.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:54.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:54.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:54.329 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:19:54.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:19:54.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:54.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:19:54.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:55.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:55.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:55.827 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:19:55.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:19:55.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:55.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:19:56.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:57.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:57.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:57.263 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:19:57.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:19:57.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:57.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:19:57.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:19:58.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:19:58.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:19:58.795 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:19:58.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:19:58.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:19:58.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:19:59.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:00.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:00.741 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:20:00.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:00.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:20:00.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:00.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:20:01.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:02.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:02.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:02.695 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:20:02.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:20:02.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:02.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:20:03.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:04.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:04.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:04.295 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:20:04.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:20:04.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:04.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:20:04.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:05.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:05.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:05.728 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:20:05.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:20:05.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:05.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:20:06.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:07.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:07.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:07.851 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:20:07.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:20:07.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:07.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:20:08.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:09.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:09.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:09.312 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:20:09.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:20:09.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:09.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:20:09.722 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2770962328,v1:127.0.0.1:6819/2770962328] [v2:127.0.0.1:6820/2770962328,v1:127.0.0.1:6821/2770962328] exists,up 8cec7fae-c64c-4f3a-900b-b8e9b009991c 2024-09-17T16:20:09.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:20:09.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:20:09.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:20:09.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:20:09.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-17T16:20:09.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:20:09.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:20:09.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T16:20:09.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:20:09.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T16:20:09.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:20:09.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:20:09.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:20:09.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:20:09.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T16:20:09.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T16:20:09.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:20:09.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:20:09.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:20:09.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:20:09.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:20:09.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:20:09.728 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:20:09.728 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:20:09.728 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:20:09.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:20:09.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:20:09.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:20:09.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:20:09.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:20:09.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:20:09.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:20:09.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:20:09.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:20:09.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:20:09.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:20:09.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:20:09.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:20:09.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T16:20:09.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:20:09.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=69e877e4-7e48-4e35-a245-6241f753c1cb 2024-09-17T16:20:09.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 69e877e4-7e48-4e35-a245-6241f753c1cb' 2024-09-17T16:20:09.730 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 69e877e4-7e48-4e35-a245-6241f753c1cb 2024-09-17T16:20:09.730 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:20:09.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA5rOlmnshGLBAAMDHpoOv9WKGieZpZ7vSFNA== 2024-09-17T16:20:09.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA5rOlmnshGLBAAMDHpoOv9WKGieZpZ7vSFNA=="}' 2024-09-17T16:20:09.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 69e877e4-7e48-4e35-a245-6241f753c1cb -i td/test-erasure-eio/3/new.json 2024-09-17T16:20:10.709 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:20:10.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-17T16:20:10.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5rOlmnshGLBAAMDHpoOv9WKGieZpZ7vSFNA== --osd-uuid 69e877e4-7e48-4e35-a245-6241f753c1cb 2024-09-17T16:20:10.751 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:10.749+0000 7f19c9e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:20:10.752 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:10.750+0000 7f19c9e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:20:10.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:10.752+0000 7f19c9e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:20:10.754 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:10.752+0000 7f19c9e0d640 -1 bdev(0x560dc3b9dc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-17T16:20:10.754 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:10.752+0000 7f19c9e0d640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-17T16:20:17.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-17T16:20:17.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:20:17.537 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-17T16:20:17.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T16:20:17.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:20:18.075 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T16:20:18.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T16:20:18.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:20:18.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:20:18.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:20:18.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:20:18.102 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:18.100+0000 7fc66f819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:20:18.116 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:18.114+0000 7fc66f819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:20:18.117 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:18.115+0000 7fc66f819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:20:18.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T16:20:18.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:20:18.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:20:18.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:20:18.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:20:18.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:18.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:20:18.498 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:20:18.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:18.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:18.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:19.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:19.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:19.933 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:20:19.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:20:19.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:19.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:19.966 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:19.964+0000 7fc66f819640 -1 Falling back to public interface 2024-09-17T16:20:20.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:21.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:21.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:21.368 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:20:21.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:20:21.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:21.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:21.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:22.797 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:22.795+0000 7fc66f819640 -1 osd.3 0 log_to_monitors true 2024-09-17T16:20:22.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:22.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:22.813 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:20:22.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:20:22.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:22.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:23.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:24.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:24.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:24.282 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:20:24.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:20:24.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:24.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:24.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:25.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:25.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:25.998 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:20:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:20:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:26.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:27.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:27.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:27.401 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:20:27.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:20:27.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:27.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:27.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:28.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:28.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:28.803 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:20:28.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:20:28.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:28.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:29.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:30.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:30.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:30.229 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:20:30.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:20:30.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:30.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:30.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:20:31.699 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:20:31.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:31.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:32.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:33.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:33.111 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:20:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:20:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:33.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:34.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:34.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:34.676 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:20:34.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:20:34.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:34.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:35.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:36.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:36.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:36.098 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:20:36.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:20:36.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:36.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:36.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:37.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:37.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:37.895 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:20:37.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:20:37.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:37.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:38.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:39.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:39.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:39.447 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:20:39.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:20:39.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:39.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:40.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:41.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:41.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:41.138 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:20:41.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:20:41.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:41.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:41.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:42.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:42.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:42.831 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:20:42.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:20:42.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:42.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:44.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:45.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:45.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:45.146 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:20:45.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:20:45.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:45.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:46.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:46.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:46.954 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:20:46.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:20:46.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:46.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:20:47.598 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2863094564,v1:127.0.0.1:6827/2863094564] [v2:127.0.0.1:6828/2863094564,v1:127.0.0.1:6829/2863094564] exists,up 69e877e4-7e48-4e35-a245-6241f753c1cb 2024-09-17T16:20:47.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:20:47.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:20:47.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:20:47.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:20:47.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-17T16:20:47.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:20:47.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:20:47.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-17T16:20:47.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:20:47.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-17T16:20:47.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:20:47.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:20:47.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:20:47.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:20:47.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-17T16:20:47.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-17T16:20:47.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:20:47.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:20:47.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:20:47.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:20:47.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:20:47.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:20:47.605 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:20:47.605 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:20:47.605 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:20:47.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:20:47.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:20:47.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:20:47.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:20:47.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:20:47.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:20:47.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:20:47.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:20:47.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:20:47.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:20:47.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:20:47.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:20:47.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:20:47.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-17T16:20:47.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:20:47.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ea681214-fe61-4ffc-ac82-0380f6822dcb 2024-09-17T16:20:47.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 ea681214-fe61-4ffc-ac82-0380f6822dcb' 2024-09-17T16:20:47.607 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 ea681214-fe61-4ffc-ac82-0380f6822dcb 2024-09-17T16:20:47.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:20:47.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBfrOlmqN/WJBAAbmtXEaBxL/KBLleryub3mg== 2024-09-17T16:20:47.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBfrOlmqN/WJBAAbmtXEaBxL/KBLleryub3mg=="}' 2024-09-17T16:20:47.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ea681214-fe61-4ffc-ac82-0380f6822dcb -i td/test-erasure-eio/4/new.json 2024-09-17T16:20:48.173 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:20:48.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-17T16:20:48.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBfrOlmqN/WJBAAbmtXEaBxL/KBLleryub3mg== --osd-uuid ea681214-fe61-4ffc-ac82-0380f6822dcb 2024-09-17T16:20:48.214 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:48.212+0000 7f1329010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:20:48.364 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:48.362+0000 7f1329010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:20:48.365 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:48.363+0000 7f1329010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:20:48.367 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:48.363+0000 7f1329010640 -1 bdev(0x558110667c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-17T16:20:48.367 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:48.364+0000 7f1329010640 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-17T16:20:54.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-17T16:20:54.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:20:54.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-17T16:20:54.550 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-17T16:20:54.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:20:55.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-17T16:20:55.091 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-17T16:20:55.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:20:55.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:20:55.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:20:55.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:20:55.118 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:55.116+0000 7fedb0410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:20:55.138 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:55.136+0000 7fedb0410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:20:55.139 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:55.137+0000 7fedb0410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:20:55.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-17T16:20:55.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:20:55.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-17T16:20:55.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:20:55.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:20:55.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:55.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:20:55.553 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:20:55.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:55.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:20:55.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:56.001 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:55.999+0000 7fedb0410640 -1 Falling back to public interface 2024-09-17T16:20:56.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:56.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:56.972 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:20:56.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:20:56.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:56.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:20:57.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:58.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:58.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:58.398 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:20:58.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:20:58.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:58.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:20:58.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:20:58.849 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:20:58.847+0000 7fedb0410640 -1 osd.4 0 log_to_monitors true 2024-09-17T16:20:59.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:20:59.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:20:59.826 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:20:59.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:20:59.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:20:59.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:00.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:01.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:01.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:01.395 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:21:01.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:21:01.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:01.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:02.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:03.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:03.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:03.110 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:21:03.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:21:03.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:03.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:03.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:04.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:04.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:04.809 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:21:04.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:21:04.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:04.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:05.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:06.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:06.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:06.330 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:21:06.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:21:06.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:06.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:06.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:07.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:07.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:07.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:21:07.751 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:21:07.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:07.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:08.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:09.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:09.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:09.171 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:21:09.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:21:09.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:09.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:09.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:10.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:10.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:10.581 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:21:10.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:21:10.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:10.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:11.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:12.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:12.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:12.045 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:21:12.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:21:12.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:12.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:12.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:13.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:13.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:13.590 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:21:13.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:21:13.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:13.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:14.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:15.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:15.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:15.317 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:21:15.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:21:15.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:15.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:16.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:17.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:17.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:17.095 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:21:17.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:21:17.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:17.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:18.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:19.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:19.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:19.248 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:21:19.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:21:19.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:19.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:20.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:21.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:21.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:21.034 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:21:21.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:21:21.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:21.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:21.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:22.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:22.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:22.953 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:21:22.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:21:22.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:22.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:23.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:24.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:24.691 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:21:24.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:24.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:21:24.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:24.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:21:25.232 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2645076149,v1:127.0.0.1:6835/2645076149] [v2:127.0.0.1:6836/2645076149,v1:127.0.0.1:6837/2645076149] exists,up ea681214-fe61-4ffc-ac82-0380f6822dcb 2024-09-17T16:21:25.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:21:25.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:21:25.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:21:25.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:21:25.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-17T16:21:25.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:21:25.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:21:25.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-17T16:21:25.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:21:25.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-17T16:21:25.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:21:25.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:21:25.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:21:25.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:21:25.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-17T16:21:25.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-17T16:21:25.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:21:25.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:21:25.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:21:25.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:21:25.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:21:25.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:21:25.237 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:21:25.237 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:21:25.238 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:21:25.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:21:25.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:21:25.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:21:25.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:21:25.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:21:25.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:21:25.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:21:25.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:21:25.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:21:25.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:21:25.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:21:25.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:21:25.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:21:25.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-17T16:21:25.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:21:25.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=906d3700-0b2f-4256-9e5c-34844d810a8c 2024-09-17T16:21:25.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 906d3700-0b2f-4256-9e5c-34844d810a8c' 2024-09-17T16:21:25.239 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 906d3700-0b2f-4256-9e5c-34844d810a8c 2024-09-17T16:21:25.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:21:25.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCFrOlma94IDxAA8ICOvwNnxd22QSqweNh8bg== 2024-09-17T16:21:25.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCFrOlma94IDxAA8ICOvwNnxd22QSqweNh8bg=="}' 2024-09-17T16:21:25.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 906d3700-0b2f-4256-9e5c-34844d810a8c -i td/test-erasure-eio/5/new.json 2024-09-17T16:21:26.098 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:21:26.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-17T16:21:26.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFrOlma94IDxAA8ICOvwNnxd22QSqweNh8bg== --osd-uuid 906d3700-0b2f-4256-9e5c-34844d810a8c 2024-09-17T16:21:26.140 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:21:26.137+0000 7fd43962e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:21:26.142 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:21:26.139+0000 7fd43962e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:21:26.143 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:21:26.140+0000 7fd43962e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:21:26.143 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:21:26.140+0000 7fd43962e640 -1 bdev(0x556fff6b5c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2024-09-17T16:21:26.143 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:21:26.140+0000 7fd43962e640 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-17T16:21:33.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-17T16:21:33.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:21:33.628 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2024-09-17T16:21:33.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-17T16:21:33.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:21:34.171 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2024-09-17T16:21:34.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-17T16:21:34.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:21:34.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:21:34.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:21:34.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:21:34.200 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:21:34.196+0000 7f5ff0e28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:21:34.213 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:21:34.210+0000 7f5ff0e28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:21:34.213 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:21:34.212+0000 7f5ff0e28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:21:34.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-17T16:21:34.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:21:34.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T16:21:34.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:21:34.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:21:34.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:34.611 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:21:34.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:21:34.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:34.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:35.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:35.573 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:21:35.571+0000 7f5ff0e28640 -1 Falling back to public interface 2024-09-17T16:21:36.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:36.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:36.015 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:21:36.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:21:36.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:36.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:36.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:37.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:37.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:37.421 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:21:37.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:21:37.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:37.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:37.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:38.534 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:21:38.532+0000 7f5ff0e28640 -1 osd.5 0 log_to_monitors true 2024-09-17T16:21:38.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:38.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:38.818 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:21:38.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:21:38.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:38.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:39.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:21:40.347 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:21:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:41.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:42.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:42.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:42.124 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:21:42.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:21:42.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:42.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:42.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:43.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:43.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:43.673 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:21:43.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:21:43.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:43.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:44.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:45.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:45.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:45.074 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:21:45.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:21:45.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:45.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:45.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:46.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:46.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:46.494 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:21:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:21:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:46.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:47.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:47.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:47.917 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:21:47.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:21:47.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:47.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:48.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:49.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:49.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:49.557 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:21:49.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:21:49.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:49.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:49.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:50.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:50.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:50.972 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:21:50.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:21:50.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:50.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:51.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:52.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:52.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:52.637 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:21:52.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:21:52.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:52.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:53.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:54.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:54.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:54.751 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:21:54.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:21:54.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:54.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:55.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:56.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:56.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:56.234 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:21:56.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:21:56.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:56.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:56.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:57.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:57.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:57.641 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:21:57.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:21:57.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:57.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:21:58.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:21:59.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:21:59.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:21:59.349 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:21:59.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:21:59.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:21:59.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:22:00.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:01.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:01.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:01.153 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:22:01.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:22:01.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:01.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:22:01.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:02.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:02.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:02.996 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:22:02.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:22:02.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:02.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:22:03.546 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1099430169,v1:127.0.0.1:6843/1099430169] [v2:127.0.0.1:6844/1099430169,v1:127.0.0.1:6845/1099430169] exists,up 906d3700-0b2f-4256-9e5c-34844d810a8c 2024-09-17T16:22:03.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:22:03.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:22:03.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:22:03.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:22:03.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-17T16:22:03.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:22:03.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:22:03.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-17T16:22:03.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:22:03.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-17T16:22:03.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:22:03.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:22:03.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:22:03.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:22:03.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-17T16:22:03.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-17T16:22:03.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:22:03.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:22:03.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:22:03.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:22:03.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:22:03.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:22:03.552 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:22:03.552 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:22:03.552 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:22:03.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:22:03.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:22:03.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:22:03.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:22:03.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:22:03.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:22:03.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:22:03.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:22:03.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:22:03.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:22:03.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:22:03.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:22:03.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:22:03.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-17T16:22:03.553 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:22:03.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b87878de-fcc3-495f-8cae-b780b4c44080 2024-09-17T16:22:03.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 b87878de-fcc3-495f-8cae-b780b4c44080' 2024-09-17T16:22:03.554 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 b87878de-fcc3-495f-8cae-b780b4c44080 2024-09-17T16:22:03.555 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:22:03.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCrrOlmuL7GIRAApi3HKi02vc3O3UDl6RoWRg== 2024-09-17T16:22:03.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCrrOlmuL7GIRAApi3HKi02vc3O3UDl6RoWRg=="}' 2024-09-17T16:22:03.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b87878de-fcc3-495f-8cae-b780b4c44080 -i td/test-erasure-eio/6/new.json 2024-09-17T16:22:04.057 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:22:04.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-17T16:22:04.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCrrOlmuL7GIRAApi3HKi02vc3O3UDl6RoWRg== --osd-uuid b87878de-fcc3-495f-8cae-b780b4c44080 2024-09-17T16:22:04.098 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:22:04.096+0000 7fc74cc44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:22:04.101 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:22:04.098+0000 7fc74cc44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:22:04.102 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:22:04.099+0000 7fc74cc44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:22:04.102 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:22:04.100+0000 7fc74cc44640 -1 bdev(0x55b637195c00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2024-09-17T16:22:04.102 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:22:04.100+0000 7fc74cc44640 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-17T16:22:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-17T16:22:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:22:10.863 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2024-09-17T16:22:10.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-17T16:22:10.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:22:11.401 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2024-09-17T16:22:11.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-17T16:22:11.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:22:11.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:22:11.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:22:11.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:22:11.429 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:22:11.426+0000 7f29abc12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:22:11.444 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:22:11.441+0000 7f29abc12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:22:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:22:11.442+0000 7f29abc12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:22:11.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-17T16:22:11.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:22:11.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-17T16:22:11.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:22:11.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:22:11.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:11.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:22:11.867 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:22:11.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:11.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:12.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:12.780 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:22:12.777+0000 7f29abc12640 -1 Falling back to public interface 2024-09-17T16:22:13.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:13.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:13.252 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:22:13.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:22:13.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:13.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:13.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:14.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:14.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:14.686 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:22:14.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:22:14.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:14.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:15.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:15.674 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:22:15.671+0000 7f29abc12640 -1 osd.6 0 log_to_monitors true 2024-09-17T16:22:16.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:16.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:16.114 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:22:16.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:22:16.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:16.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:16.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:17.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:17.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:17.976 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:22:17.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:22:17.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:17.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:18.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:19.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:19.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:19.791 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:22:19.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:22:19.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:19.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:20.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:21.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:21.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:21.209 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:22:21.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:22:21.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:21.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:21.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:22.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:22.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:22.649 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:22:22.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:22:22.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:22.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:23.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:24.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:24.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:24.057 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:22:24.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:22:24.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:24.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:24.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:25.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:25.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:25.548 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:22:25.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:22:25.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:25.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:25.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:26.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:26.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:26.972 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:22:26.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:22:26.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:26.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:27.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:28.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:28.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:28.514 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:22:28.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:22:28.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:28.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:28.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:29.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:29.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:29.943 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:22:29.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:22:29.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:29.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:30.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:31.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:31.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:31.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:22:31.622 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:22:31.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:31.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:32.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:33.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:33.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:33.459 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:22:33.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:22:33.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:33.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:34.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:35.245 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:22:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:22:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:35.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:36.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:37.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:37.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:37.141 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:22:37.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:22:37.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:37.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:37.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:38.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:38.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:38.743 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:22:38.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:22:38.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:38.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:39.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:40.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:40.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:40.508 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:22:40.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:22:40.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:40.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:22:41.029 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1470707139,v1:127.0.0.1:6851/1470707139] [v2:127.0.0.1:6852/1470707139,v1:127.0.0.1:6853/1470707139] exists,up b87878de-fcc3-495f-8cae-b780b4c44080 2024-09-17T16:22:41.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:22:41.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:22:41.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:22:41.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-17T16:22:41.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:22:41.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:22:41.031 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:22:41.031 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:22:41.031 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:22:41.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T16:22:41.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-17T16:22:41.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.0.asok log flush 2024-09-17T16:22:41.110 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:22:41.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-17T16:22:41.122 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:18:31.733+0000 7f4f3a032640 0 load: jerasure load: lrc 2024-09-17T16:22:41.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:384: TEST_ec_object_attr_read_error: local poolname=pool-jerasure 2024-09-17T16:22:41.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:385: TEST_ec_object_attr_read_error: create_erasure_coded_pool pool-jerasure 3 2 2024-09-17T16:22:41.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:22:41.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-17T16:22:41.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-17T16:22:41.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-17T16:22:41.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-17T16:22:41.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-17T16:22:41.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-17T16:22:41.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-17T16:22:41.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-17T16:22:43.298 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-17T16:22:43.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:22:44.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-17T16:22:44.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:22:44.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:22:44.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:22:44.312 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:22:44.312 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:22:44.312 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:22:44.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:22:44.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:22:44.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:22:44.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:22:44.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:22:44.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:22:44.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:22:44.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:22:44.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:22:44.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:22:44.947 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:22:44.947 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:22:44.948 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:22:44.948 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:22:44.948 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:22:44.948 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-17T16:22:44.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:22:44.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:22:44.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:22:45.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803829 2024-09-17T16:22:45.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803829 2024-09-17T16:22:45.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829' 2024-09-17T16:22:45.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:22:45.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:22:45.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=60129542188 2024-09-17T16:22:45.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 60129542188 2024-09-17T16:22:45.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829 1-60129542188' 2024-09-17T16:22:45.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:22:45.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:22:45.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345958 2024-09-17T16:22:45.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345958 2024-09-17T16:22:45.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829 1-60129542188 2-85899345958' 2024-09-17T16:22:45.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:22:45.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:22:45.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018907 2024-09-17T16:22:45.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018907 2024-09-17T16:22:45.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829 1-60129542188 2-85899345958 3-128849018907' 2024-09-17T16:22:45.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:22:45.577 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:22:45.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691865 2024-09-17T16:22:45.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691865 2024-09-17T16:22:45.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829 1-60129542188 2-85899345958 3-128849018907 4-171798691865' 2024-09-17T16:22:45.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:22:45.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:22:45.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332112 2024-09-17T16:22:45.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332112 2024-09-17T16:22:45.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829 1-60129542188 2-85899345958 3-128849018907 4-171798691865 5-219043332112' 2024-09-17T16:22:45.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:22:45.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:22:45.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=261993005063 2024-09-17T16:22:45.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 261993005063 2024-09-17T16:22:45.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829 1-60129542188 2-85899345958 3-128849018907 4-171798691865 5-219043332112 6-261993005063' 2024-09-17T16:22:45.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:22:45.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803829 2024-09-17T16:22:45.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:22:45.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:22:45.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803829 2024-09-17T16:22:45.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:22:45.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803829 2024-09-17T16:22:45.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803829' 2024-09-17T16:22:45.921 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803829 2024-09-17T16:22:45.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:22:46.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803829 -lt 25769803829 2024-09-17T16:22:46.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:22:46.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-60129542188 2024-09-17T16:22:46.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:22:46.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:22:46.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-60129542188 2024-09-17T16:22:46.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:22:46.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=60129542188 2024-09-17T16:22:46.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 60129542188' 2024-09-17T16:22:46.319 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 60129542188 2024-09-17T16:22:46.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:22:46.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 60129542188 -lt 60129542188 2024-09-17T16:22:46.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:22:46.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345958 2024-09-17T16:22:46.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:22:46.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:22:46.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345958 2024-09-17T16:22:46.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:22:46.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345958 2024-09-17T16:22:46.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345958' 2024-09-17T16:22:46.750 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 85899345958 2024-09-17T16:22:46.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:22:47.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345958 -lt 85899345958 2024-09-17T16:22:47.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:22:47.138 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018907 2024-09-17T16:22:47.138 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:22:47.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:22:47.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018907 2024-09-17T16:22:47.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:22:47.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018907 2024-09-17T16:22:47.140 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 128849018907 2024-09-17T16:22:47.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018907' 2024-09-17T16:22:47.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:22:47.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018907 -lt 128849018907 2024-09-17T16:22:47.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:22:47.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691865 2024-09-17T16:22:47.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:22:47.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:22:47.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691865 2024-09-17T16:22:47.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:22:47.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691865 2024-09-17T16:22:47.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691865' 2024-09-17T16:22:47.562 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691865 2024-09-17T16:22:47.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:22:47.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691865 -lt 171798691865 2024-09-17T16:22:47.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:22:47.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-219043332112 2024-09-17T16:22:47.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:22:47.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:22:47.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-219043332112 2024-09-17T16:22:47.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:22:47.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332112 2024-09-17T16:22:47.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 219043332112' 2024-09-17T16:22:47.988 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 219043332112 2024-09-17T16:22:47.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:22:48.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332113 -lt 219043332112 2024-09-17T16:22:48.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:22:48.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-261993005063 2024-09-17T16:22:48.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:22:48.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:22:48.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-261993005063 2024-09-17T16:22:48.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:22:48.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=261993005063 2024-09-17T16:22:48.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 261993005063' 2024-09-17T16:22:48.390 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 261993005063 2024-09-17T16:22:48.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:22:48.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 261993005064 -lt 261993005063 2024-09-17T16:22:48.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:22:48.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:22:48.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:22:49.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:22:49.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:22:49.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:22:49.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:22:49.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:22:49.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:22:49.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:22:49.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:22:49.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:22:49.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:22:49.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:22:49.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:22:50.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:22:50.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:22:50.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:22:50.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:387: TEST_ec_object_attr_read_error: get_primary pool-jerasure myobject 2024-09-17T16:22:50.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=pool-jerasure 2024-09-17T16:22:50.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=myobject 2024-09-17T16:22:50.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map pool-jerasure myobject 2024-09-17T16:22:50.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T16:22:50.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:387: TEST_ec_object_attr_read_error: local primary_osd=3 2024-09-17T16:22:50.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:389: TEST_ec_object_attr_read_error: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-17T16:22:50.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:22:50.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:22:50.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:22:50.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:22:50.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:22:51.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:22:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:392: TEST_ec_object_attr_read_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-17T16:22:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:22:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:22:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-17T16:22:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:22:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:22:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:22:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:22:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:22:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:22:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:22:51.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:22:51.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-17T16:22:52.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:395: TEST_ec_object_attr_read_error: inject_eio ec mdata pool-jerasure myobject td/test-erasure-eio 1 2024-09-17T16:22:52.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T16:22:52.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T16:22:52.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-17T16:22:52.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T16:22:52.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T16:22:52.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T16:22:52.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-17T16:22:52.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T16:22:52.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T16:22:52.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T16:22:52.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-17T16:22:52.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T16:22:52.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-17T16:22:52.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:22:52.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-17T16:22:52.413 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-17T16:22:52.413 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:22:52.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2147483647 2024-09-17T16:22:52.814 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:22:52.814 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:22:52.814 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:22:52.814 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-17T16:22:52.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2147483647 5 1 0 6 2024-09-17T16:22:52.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('2147483647' '5' '1' '0' '6') 2024-09-17T16:22:52.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T16:22:52.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=5 2024-09-17T16:22:52.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T16:22:52.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/5/type 2024-09-17T16:22:52.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T16:22:52.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 5 bluestore_debug_inject_read_err true 2024-09-17T16:22:52.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T16:22:52.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=5 2024-09-17T16:22:52.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T16:22:52.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T16:22:52.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T16:22:52.817 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.5 2024-09-17T16:22:52.817 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.5 2024-09-17T16:22:52.817 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.5 ']' 2024-09-17T16:22:52.817 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:22:52.817 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:22:52.817 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:22:52.817 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.5.asok 2024-09-17T16:22:52.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.5.asok config set bluestore_debug_inject_read_err true 2024-09-17T16:22:52.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T16:22:52.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T16:22:52.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T16:22:52.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.5 2024-09-17T16:22:52.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.5 2024-09-17T16:22:52.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.5 ']' 2024-09-17T16:22:52.904 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:22:52.904 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:22:52.904 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:22:52.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.5.asok 2024-09-17T16:22:52.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T16:22:52.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.5.asok injectmdataerr pool-jerasure myobject 1 2024-09-17T16:22:52.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:398: TEST_ec_object_attr_read_error: activate_osd td/test-erasure-eio 3 2024-09-17T16:22:52.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:22:52.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:22:52.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T16:22:52.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:22:52.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T16:22:52.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:22:52.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:22:52.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:22:52.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:22:52.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T16:22:52.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T16:22:52.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:22:52.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:22:52.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:22:52.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:22:52.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:22:52.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:22:52.996 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:22:52.996 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:22:52.996 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:22:52.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:22:52.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:22:52.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:22:52.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:22:52.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:22:52.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:22:52.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:22:52.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:22:52.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:22:52.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:22:52.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:22:52.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T16:22:52.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T16:22:52.997 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T16:22:52.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:22:52.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-17T16:22:52.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T16:22:52.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:22:52.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:22:52.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:22:53.021 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:22:53.019+0000 7fb2e3664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:22:53.023 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:22:53.021+0000 7fb2e3664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:22:53.024 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:22:53.022+0000 7fb2e3664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:22:53.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T16:22:53.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:22:53.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:22:53.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:22:53.426 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:22:53.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:22:53.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:53.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:22:53.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:53.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:22:53.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:54.387 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:22:54.385+0000 7fb2e3664640 -1 Falling back to public interface 2024-09-17T16:22:54.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:54.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:54.815 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:22:54.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:22:54.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:54.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:22:55.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:56.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:56.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:56.257 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:22:56.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:22:56.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:56.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:22:56.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:22:57.502 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:22:57.499+0000 7fb2e3664640 -1 osd.3 69 log_to_monitors true 2024-09-17T16:22:57.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:22:57.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:22:57.665 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:22:57.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:22:57.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:22:57.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:22:58.156 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 73 up_thru 66 down_at 70 last_clean_interval [30,69) [v2:127.0.0.1:6826/2909736754,v1:127.0.0.1:6827/2909736754] [v2:127.0.0.1:6828/2909736754,v1:127.0.0.1:6829/2909736754] exists,up 69e877e4-7e48-4e35-a245-6241f753c1cb 2024-09-17T16:22:58.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:22:58.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:22:58.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:22:58.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_object_attr_read_error: wait_for_clean 2024-09-17T16:22:58.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:22:58.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:22:58.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:22:58.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:22:58.158 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:22:58.158 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:22:58.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:22:58.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:22:58.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:22:58.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:22:58.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:22:58.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:22:58.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:22:58.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:22:58.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:22:58.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:22:58.746 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:22:58.747 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:22:58.747 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:22:58.747 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:22:58.747 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:22:58.747 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-17T16:22:58.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:22:58.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:22:58.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:22:58.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803834 2024-09-17T16:22:58.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803834 2024-09-17T16:22:58.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803834' 2024-09-17T16:22:58.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:22:58.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:22:59.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=60129542193 2024-09-17T16:22:59.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 60129542193 2024-09-17T16:22:59.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803834 1-60129542193' 2024-09-17T16:22:59.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:22:59.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:22:59.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345963 2024-09-17T16:22:59.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345963 2024-09-17T16:22:59.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803834 1-60129542193 2-85899345963' 2024-09-17T16:22:59.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:22:59.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:22:59.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612611 2024-09-17T16:22:59.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612611 2024-09-17T16:22:59.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803834 1-60129542193 2-85899345963 3-313532612611' 2024-09-17T16:22:59.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:22:59.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:22:59.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691869 2024-09-17T16:22:59.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691869 2024-09-17T16:22:59.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803834 1-60129542193 2-85899345963 3-313532612611 4-171798691869' 2024-09-17T16:22:59.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:22:59.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:22:59.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332117 2024-09-17T16:22:59.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332117 2024-09-17T16:22:59.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803834 1-60129542193 2-85899345963 3-313532612611 4-171798691869 5-219043332117' 2024-09-17T16:22:59.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:22:59.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:22:59.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=261993005068 2024-09-17T16:22:59.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 261993005068 2024-09-17T16:22:59.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803834 1-60129542193 2-85899345963 3-313532612611 4-171798691869 5-219043332117 6-261993005068' 2024-09-17T16:22:59.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:22:59.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803834 2024-09-17T16:22:59.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:22:59.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:22:59.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803834 2024-09-17T16:22:59.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:22:59.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803834 2024-09-17T16:22:59.639 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803834 2024-09-17T16:22:59.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803834' 2024-09-17T16:22:59.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:23:00.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803832 -lt 25769803834 2024-09-17T16:23:00.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:23:01.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:23:01.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:23:01.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803834 -lt 25769803834 2024-09-17T16:23:01.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:23:01.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-60129542193 2024-09-17T16:23:01.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:23:01.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:23:01.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-60129542193 2024-09-17T16:23:01.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:23:01.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=60129542193 2024-09-17T16:23:01.464 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 60129542193 2024-09-17T16:23:01.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 60129542193' 2024-09-17T16:23:01.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:23:01.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 60129542193 -lt 60129542193 2024-09-17T16:23:01.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:23:01.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345963 2024-09-17T16:23:01.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:23:01.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:23:01.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345963 2024-09-17T16:23:01.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:23:01.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345963 2024-09-17T16:23:01.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345963' 2024-09-17T16:23:01.863 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 85899345963 2024-09-17T16:23:01.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:23:02.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345963 -lt 85899345963 2024-09-17T16:23:02.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:23:02.312 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612611 2024-09-17T16:23:02.312 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:23:02.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:23:02.314 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612611 2024-09-17T16:23:02.314 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:23:02.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612611 2024-09-17T16:23:02.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612611' 2024-09-17T16:23:02.315 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 313532612611 2024-09-17T16:23:02.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:23:02.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612611 -lt 313532612611 2024-09-17T16:23:02.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:23:02.775 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691869 2024-09-17T16:23:02.775 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:23:02.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:23:02.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691869 2024-09-17T16:23:02.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:23:02.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691869 2024-09-17T16:23:02.777 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691869 2024-09-17T16:23:02.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691869' 2024-09-17T16:23:02.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:23:03.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691870 -lt 171798691869 2024-09-17T16:23:03.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:23:03.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-219043332117 2024-09-17T16:23:03.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:23:03.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:23:03.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-219043332117 2024-09-17T16:23:03.166 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:23:03.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332117 2024-09-17T16:23:03.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 219043332117' 2024-09-17T16:23:03.167 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 219043332117 2024-09-17T16:23:03.167 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:23:03.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332117 -lt 219043332117 2024-09-17T16:23:03.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:23:03.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-261993005068 2024-09-17T16:23:03.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:23:03.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:23:03.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-261993005068 2024-09-17T16:23:03.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:23:03.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=261993005068 2024-09-17T16:23:03.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 261993005068' 2024-09-17T16:23:03.555 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 261993005068 2024-09-17T16:23:03.555 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:23:03.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 261993005069 -lt 261993005068 2024-09-17T16:23:03.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:23:03.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:23:03.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:23:04.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:23:04.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:23:04.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:23:04.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:23:04.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:23:04.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:23:04.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:23:04.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:23:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:23:04.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:23:04.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:23:04.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:23:05.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:23:05.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:23:05.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:23:05.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:403: TEST_ec_object_attr_read_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-17T16:23:05.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:23:05.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:23:05.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-17T16:23:05.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:23:05.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:23:05.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-17T16:23:05.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:23:05.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:23:05.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:405: TEST_ec_object_attr_read_error: delete_erasure_coded_pool pool-jerasure 2024-09-17T16:23:05.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:23:05.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-17T16:23:06.046 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-17T16:23:06.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-17T16:23:06.634 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-17T16:23:06.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-17T16:23:06.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:23:06.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:23:06.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:23:06.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:23:06.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:23:06.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:23:06.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:23:06.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:23:06.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:23:06.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:23:06.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:23:06.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:23:06.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:23:06.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:23:06.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:23:06.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:23:06.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:23:06.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:23:06.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:23:06.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:23:06.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:23:06.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:23:06.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:23:06.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:23:06.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:23:06.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:23:06.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:23:06.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:23:06.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-17T16:23:06.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-17T16:23:06.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-17T16:23:06.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-17T16:23:06.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:23:06.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:23:06.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:23:06.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:23:06.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:23:06.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:23:06.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:23:06.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:23:06.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:23:06.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:23:06.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:23:06.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:23:06.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:23:06.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:23:06.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:23:06.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:23:06.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:23:06.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:23:06.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:23:06.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:23:06.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:23:06.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:23:06.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:23:06.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:23:06.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:23:06.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:23:06.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:23:06.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:23:06.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-17T16:23:06.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:23:06.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:23:06.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:23:06.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.87940 2024-09-17T16:23:06.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:23:06.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T16:23:06.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:23:06.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-17T16:23:06.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-17T16:23:06.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-17T16:23:06.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:23:06.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:23:06.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:23:06.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-17T16:23:06.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-17T16:23:07.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:23:07.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:23:07.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:23:07.277 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:23:07.277 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:23:07.277 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:23:07.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:23:07.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T16:23:07.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:23:07.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:23:07.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:23:07.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:23:07.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:23:07.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:23:07.521 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:23:07.521 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:23:07.522 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:23:07.522 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:23:07.522 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:23:07.522 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:23:07.522 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:23:07.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:23:07.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get fsid 2024-09-17T16:23:07.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:23:07.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:23:07.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:23:07.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:23:07.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:23:07.604 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:23:07.604 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:23:07.604 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:23:07.604 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:23:07.604 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:23:07.604 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:23:07.604 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:23:07.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:23:07.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get mon_host 2024-09-17T16:23:07.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-17T16:23:07.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-17T16:23:07.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:23:07.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:23:07.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:23:07.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-17T16:23:07.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:23:08.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:23:08.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:23:08.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:23:08.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:23:08.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:23:08.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:23:08.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:23:08.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:23:08.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:23:08.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-17T16:23:08.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T16:23:08.843 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T16:23:08.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:23:09.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-17T16:23:09.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:23:09.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:23:09.855 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:23:09.855 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:23:09.855 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:23:09.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:23:09.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-17T16:23:09.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-mon.a.asok log flush 2024-09-17T16:23:09.930 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:23:09.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-17T16:23:09.938 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:23:07.308+0000 7f422100cc80 0 load: jerasure load: lrc 2024-09-17T16:23:09.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_errors td/test-erasure-eio 2024-09-17T16:23:09.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:438: TEST_ec_recovery_multiple_errors: local dir=td/test-erasure-eio 2024-09-17T16:23:09.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:439: TEST_ec_recovery_multiple_errors: local objname=myobject 2024-09-17T16:23:09.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:441: TEST_ec_recovery_multiple_errors: setup_osds 9 2024-09-17T16:23:09.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=9 2024-09-17T16:23:09.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-17T16:23:09.940 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 9 - 1 2024-09-17T16:23:09.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 8 2024-09-17T16:23:09.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:23:09.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-17T16:23:09.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:23:09.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:23:09.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:23:09.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:23:09.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-17T16:23:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:23:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:23:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:23:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:23:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-17T16:23:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-17T16:23:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:23:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:23:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:23:09.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:23:09.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:23:09.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:23:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:23:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:23:09.946 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:23:09.946 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:23:09.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:23:09.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:23:09.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:23:09.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:23:09.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:23:09.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:23:09.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:23:09.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:23:09.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:23:09.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:23:09.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:23:09.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:23:09.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-17T16:23:09.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:23:09.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e8de4132-27ce-46d2-8c56-ab79c005445d 2024-09-17T16:23:09.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e8de4132-27ce-46d2-8c56-ab79c005445d' 2024-09-17T16:23:09.947 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 e8de4132-27ce-46d2-8c56-ab79c005445d 2024-09-17T16:23:09.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:23:09.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDtrOlms/JcORAASw7fkOwvqDfPXDCYpiZQig== 2024-09-17T16:23:09.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDtrOlms/JcORAASw7fkOwvqDfPXDCYpiZQig=="}' 2024-09-17T16:23:09.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e8de4132-27ce-46d2-8c56-ab79c005445d -i td/test-erasure-eio/0/new.json 2024-09-17T16:23:10.219 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:23:10.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-17T16:23:10.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDtrOlms/JcORAASw7fkOwvqDfPXDCYpiZQig== --osd-uuid e8de4132-27ce-46d2-8c56-ab79c005445d 2024-09-17T16:23:10.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:10.258+0000 7f083fb29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:23:10.288 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:10.286+0000 7f083fb29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:23:10.289 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:10.287+0000 7f083fb29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:23:10.290 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:10.288+0000 7f083fb29640 -1 bdev(0x557c0ba15c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:23:10.290 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:10.288+0000 7f083fb29640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-17T16:23:16.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-17T16:23:16.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:23:16.402 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:23:16.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:23:16.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:23:16.939 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:23:16.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:23:16.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:23:16.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:23:16.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:23:16.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:23:16.967 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:16.965+0000 7f8274bb9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:23:16.996 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:16.994+0000 7f8274bb9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:23:16.997 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:16.995+0000 7f8274bb9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:23:17.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:23:17.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:23:17.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:23:17.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:23:17.368 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:23:17.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:23:17.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:17.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:23:17.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:17.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:17.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:18.347 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:18.345+0000 7f8274bb9640 -1 Falling back to public interface 2024-09-17T16:23:18.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:18.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:18.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:23:18.787 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:23:18.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:18.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:19.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:20.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:20.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:20.180 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:23:20.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:23:20.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:20.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:20.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:21.224 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:21.222+0000 7f8274bb9640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:23:21.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:21.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:21.572 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:23:21.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:23:21.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:21.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:21.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:22.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:22.975 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:23:22.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:22.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:23:22.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:22.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:23.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:24.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:24.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:24.375 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:23:24.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:23:24.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:24.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:24.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:25.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:25.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:25.775 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:23:25.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:23:25.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:25.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:26.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:27.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:27.407 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:23:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:23:27.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:27.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:27.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:28.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:28.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:28.809 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:23:28.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:23:28.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:28.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:29.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:30.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:30.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:30.278 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:23:30.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:23:30.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:30.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:30.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:31.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:31.685 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:23:31.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:31.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:23:31.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:31.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:32.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:33.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:33.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:33.149 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:23:33.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:23:33.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:33.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:33.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:34.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:34.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:34.941 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:23:34.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:23:34.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:34.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:35.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:36.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:36.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:36.731 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:23:36.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:23:36.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:36.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:37.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:38.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:38.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:38.214 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:23:38.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:23:38.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:38.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:38.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:39.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:39.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:39.895 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:23:39.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:23:39.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:39.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:40.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:41.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:41.541 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:23:41.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:41.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:23:41.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:41.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:42.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:43.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:43.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:43.361 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:23:43.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:23:43.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:43.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:23:43.922 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2966804186,v1:127.0.0.1:6803/2966804186] [v2:127.0.0.1:6804/2966804186,v1:127.0.0.1:6805/2966804186] exists,up e8de4132-27ce-46d2-8c56-ab79c005445d 2024-09-17T16:23:43.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:23:43.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:23:43.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:23:43.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:23:43.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-17T16:23:43.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:23:43.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:23:43.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T16:23:43.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:23:43.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:23:43.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:23:43.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:23:43.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:23:43.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:23:43.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:23:43.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:23:43.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:23:43.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:23:43.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:23:43.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:23:43.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:23:43.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:23:43.929 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:23:43.929 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:23:43.929 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:23:43.930 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:23:43.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:23:43.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:23:43.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:23:43.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:23:43.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:23:43.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:23:43.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:23:43.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:23:43.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:23:43.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:23:43.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:23:43.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:23:43.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:23:43.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:23:43.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7889f195-8930-4b2f-b6eb-510737c4cbd9 2024-09-17T16:23:43.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7889f195-8930-4b2f-b6eb-510737c4cbd9' 2024-09-17T16:23:43.932 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 7889f195-8930-4b2f-b6eb-510737c4cbd9 2024-09-17T16:23:43.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:23:43.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAPrelm9MwnOBAA6ZqTjtxqLTX+I9SQ6PsGWA== 2024-09-17T16:23:43.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAPrelm9MwnOBAA6ZqTjtxqLTX+I9SQ6PsGWA=="}' 2024-09-17T16:23:43.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7889f195-8930-4b2f-b6eb-510737c4cbd9 -i td/test-erasure-eio/1/new.json 2024-09-17T16:23:44.541 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:23:44.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-17T16:23:44.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPrelm9MwnOBAA6ZqTjtxqLTX+I9SQ6PsGWA== --osd-uuid 7889f195-8930-4b2f-b6eb-510737c4cbd9 2024-09-17T16:23:44.582 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:44.579+0000 7f19dca10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:23:44.599 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:44.596+0000 7f19dca10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:23:44.600 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:44.597+0000 7f19dca10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:23:44.600 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:44.597+0000 7f19dca10640 -1 bdev(0x55eadad45c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-17T16:23:44.600 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:44.598+0000 7f19dca10640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-17T16:23:50.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-17T16:23:50.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:23:50.892 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T16:23:50.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T16:23:50.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:23:51.432 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T16:23:51.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T16:23:51.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:23:51.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:23:51.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:23:51.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:23:51.460 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:51.457+0000 7fc701d30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:23:51.507 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:51.504+0000 7fc701d30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:23:51.508 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:51.505+0000 7fc701d30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:23:51.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T16:23:51.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:23:51.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:23:51.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:23:51.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:23:51.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:51.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:23:51.847 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:23:51.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:51.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:23:52.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:52.850 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:52.847+0000 7fc701d30640 -1 Falling back to public interface 2024-09-17T16:23:53.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:53.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:53.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:23:53.274 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:23:53.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:53.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:23:53.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:54.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:54.710 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:23:54.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:54.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:23:54.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:54.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:23:55.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:55.863 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:23:55.861+0000 7fc701d30640 -1 osd.1 0 log_to_monitors true 2024-09-17T16:23:56.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:56.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:56.136 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:23:56.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:23:56.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:56.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:23:56.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:57.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:57.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:57.682 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:23:57.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:23:57.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:57.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:23:58.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:23:59.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:23:59.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:23:59.174 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:23:59.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:23:59.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:23:59.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:23:59.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:00.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:00.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:00.660 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:24:00.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:24:00.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:00.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:24:01.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:02.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:02.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:02.139 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:24:02.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:24:02.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:02.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:24:02.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:03.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:03.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:03.537 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:24:03.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:24:03.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:03.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:24:04.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:05.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:05.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:05.113 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:24:05.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:24:05.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:05.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:24:05.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:06.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:06.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:06.534 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:24:06.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:24:06.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:06.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:24:07.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:08.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:08.008 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:24:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:24:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:24:08.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:09.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:09.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:09.609 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:24:09.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:24:09.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:09.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:24:10.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:11.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:11.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:11.629 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:24:11.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:24:11.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:11.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:24:12.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:13.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:13.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:13.454 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:24:13.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:24:13.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:13.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:24:14.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:15.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:15.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:15.187 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:24:15.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:24:15.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:15.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:24:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:16.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:16.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:16.868 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:24:16.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:24:16.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:16.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:24:17.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:18.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:18.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:18.763 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:24:18.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:24:18.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:18.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:24:19.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:20.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:20.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:20.256 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:24:20.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:24:20.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:20.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:24:20.666 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2377554229,v1:127.0.0.1:6811/2377554229] [v2:127.0.0.1:6812/2377554229,v1:127.0.0.1:6813/2377554229] exists,up 7889f195-8930-4b2f-b6eb-510737c4cbd9 2024-09-17T16:24:20.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:24:20.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:24:20.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:24:20.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:24:20.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-17T16:24:20.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:24:20.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:24:20.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T16:24:20.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:24:20.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:24:20.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:24:20.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:24:20.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:24:20.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:24:20.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:24:20.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:24:20.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:24:20.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:24:20.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:24:20.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:24:20.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:24:20.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:24:20.671 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:24:20.671 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:24:20.671 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:24:20.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:24:20.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:24:20.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:24:20.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:24:20.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:24:20.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:24:20.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:24:20.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:24:20.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:24:20.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:24:20.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:24:20.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:24:20.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:24:20.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:24:20.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:24:20.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1439848f-7801-43b9-9ce9-c9b07de36e06 2024-09-17T16:24:20.673 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 1439848f-7801-43b9-9ce9-c9b07de36e06 2024-09-17T16:24:20.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 1439848f-7801-43b9-9ce9-c9b07de36e06' 2024-09-17T16:24:20.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:24:20.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA0relmf/3jKBAA9Fm75K40Tnzuoaw2sgJ8jQ== 2024-09-17T16:24:20.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA0relmf/3jKBAA9Fm75K40Tnzuoaw2sgJ8jQ=="}' 2024-09-17T16:24:20.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1439848f-7801-43b9-9ce9-c9b07de36e06 -i td/test-erasure-eio/2/new.json 2024-09-17T16:24:21.189 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:24:21.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-17T16:24:21.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA0relmf/3jKBAA9Fm75K40Tnzuoaw2sgJ8jQ== --osd-uuid 1439848f-7801-43b9-9ce9-c9b07de36e06 2024-09-17T16:24:21.230 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:24:21.228+0000 7f113a22f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:24:21.257 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:24:21.255+0000 7f113a22f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:24:21.258 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:24:21.256+0000 7f113a22f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:24:21.259 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:24:21.257+0000 7f113a22f640 -1 bdev(0x55d8c916fc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-17T16:24:21.259 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:24:21.257+0000 7f113a22f640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-17T16:24:27.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-17T16:24:27.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:24:27.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T16:24:27.815 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T16:24:27.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:24:28.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T16:24:28.356 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T16:24:28.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:24:28.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:24:28.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:24:28.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:24:28.383 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:24:28.381+0000 7f232d0e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:24:28.397 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:24:28.395+0000 7f232d0e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:24:28.398 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:24:28.396+0000 7f232d0e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:24:28.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T16:24:28.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:24:28.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:24:28.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:24:28.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:24:28.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:28.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:24:28.784 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:24:28.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:28.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:29.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:29.494 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:24:29.492+0000 7f232d0e2640 -1 Falling back to public interface 2024-09-17T16:24:30.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:30.214 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:24:30.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:30.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:24:30.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:30.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:30.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:31.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:31.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:31.641 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:24:31.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:24:31.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:31.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:31.959 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:24:31.956+0000 7f232d0e2640 -1 osd.2 0 log_to_monitors true 2024-09-17T16:24:32.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:33.037 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:24:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:24:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:33.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:34.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:34.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:34.478 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:24:34.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:24:34.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:34.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:35.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:36.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:36.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:36.077 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:24:36.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:24:36.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:36.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:36.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:37.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:37.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:37.487 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:24:37.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:24:37.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:37.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:38.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:39.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:39.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:39.027 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:24:39.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:24:39.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:39.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:39.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:40.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:40.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:40.448 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:24:40.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:24:40.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:40.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:40.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:41.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:41.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:41.981 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:24:41.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:24:41.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:41.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:42.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:43.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:43.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:43.384 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:24:43.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:24:43.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:43.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:43.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:44.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:44.889 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:24:44.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:44.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:24:44.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:44.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:45.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:46.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:46.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:46.880 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:24:46.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:24:46.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:46.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:47.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:48.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:48.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:48.901 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:24:48.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:24:48.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:48.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:49.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:50.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:50.836 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:24:50.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:50.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:24:50.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:50.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:51.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:52.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:52.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:52.892 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:24:52.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:24:52.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:52.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:54.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:55.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:55.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:55.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:24:55.029 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:24:55.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:55.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:55.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:24:56.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:24:56.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:24:56.529 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:24:56.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:24:56.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:24:56.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:24:57.057 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/725596860,v1:127.0.0.1:6819/725596860] [v2:127.0.0.1:6820/725596860,v1:127.0.0.1:6821/725596860] exists,up 1439848f-7801-43b9-9ce9-c9b07de36e06 2024-09-17T16:24:57.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:24:57.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:24:57.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:24:57.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:24:57.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-17T16:24:57.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:24:57.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:24:57.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T16:24:57.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:24:57.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T16:24:57.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:24:57.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:24:57.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:24:57.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:24:57.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T16:24:57.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T16:24:57.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:24:57.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:24:57.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:24:57.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:24:57.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:24:57.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:24:57.063 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:24:57.063 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:24:57.063 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:24:57.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:24:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:24:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:24:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:24:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:24:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:24:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:24:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:24:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:24:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:24:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:24:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:24:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:24:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T16:24:57.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:24:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5af29edd-1ab3-4a77-8e37-d19e41affdd2 2024-09-17T16:24:57.065 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 5af29edd-1ab3-4a77-8e37-d19e41affdd2 2024-09-17T16:24:57.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 5af29edd-1ab3-4a77-8e37-d19e41affdd2' 2024-09-17T16:24:57.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:24:57.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBZrelmdxaZBBAAJga8H5txIhUQ8PV2zVWK9Q== 2024-09-17T16:24:57.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBZrelmdxaZBBAAJga8H5txIhUQ8PV2zVWK9Q=="}' 2024-09-17T16:24:57.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5af29edd-1ab3-4a77-8e37-d19e41affdd2 -i td/test-erasure-eio/3/new.json 2024-09-17T16:24:57.614 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:24:57.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-17T16:24:57.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZrelmdxaZBBAAJga8H5txIhUQ8PV2zVWK9Q== --osd-uuid 5af29edd-1ab3-4a77-8e37-d19e41affdd2 2024-09-17T16:24:57.657 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:24:57.654+0000 7f56938ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:24:57.689 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:24:57.687+0000 7f56938ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:24:57.690 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:24:57.688+0000 7f56938ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:24:57.690 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:24:57.689+0000 7f56938ae640 -1 bdev(0x5624b2297c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-17T16:24:57.690 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:24:57.689+0000 7f56938ae640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-17T16:25:03.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-17T16:25:03.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:25:03.709 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-17T16:25:03.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T16:25:03.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:25:04.250 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T16:25:04.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T16:25:04.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:25:04.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:25:04.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:25:04.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:25:04.278 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:25:04.275+0000 7fefaa5b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:25:04.294 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:25:04.292+0000 7fefaa5b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:25:04.295 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:25:04.293+0000 7fefaa5b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:25:04.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T16:25:04.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:25:04.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:25:04.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:25:04.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:25:04.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:04.741 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:25:04.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:25:04.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:04.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:05.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:05.902 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:25:05.900+0000 7fefaa5b2640 -1 Falling back to public interface 2024-09-17T16:25:06.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:06.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:06.166 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:25:06.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:25:06.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:06.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:06.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:07.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:07.605 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:25:07.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:07.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:25:07.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:07.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:08.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:08.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:25:08.473+0000 7fefaa5b2640 -1 osd.3 0 log_to_monitors true 2024-09-17T16:25:09.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:09.041 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:25:09.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:09.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:25:09.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:09.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:09.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:10.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:10.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:10.733 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:25:10.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:25:10.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:10.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:11.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:12.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:12.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:12.175 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:25:12.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:25:12.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:12.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:12.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:13.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:13.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:13.682 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:25:13.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:25:13.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:13.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:14.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:15.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:15.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:15.211 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:25:15.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:25:15.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:15.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:15.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:16.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:16.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:16.623 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:25:16.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:25:16.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:16.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:17.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:18.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:18.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:18.052 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:25:18.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:25:18.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:18.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:18.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:19.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:19.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:19.453 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:25:19.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:25:19.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:19.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:19.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:20.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:20.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:20.913 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:25:20.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:25:20.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:20.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:21.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:22.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:22.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:22.443 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:25:22.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:25:22.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:22.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:23.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:24.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:24.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:24.131 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:25:24.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:25:24.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:24.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:24.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:25.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:25.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:25.808 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:25:25.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:25:25.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:25.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:26.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:27.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:27.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:27.586 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:25:27.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:25:27.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:27.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:28.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:29.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:29.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:29.456 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:25:29.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:25:29.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:29.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:30.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:31.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:31.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:31.442 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:25:31.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:25:31.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:31.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:32.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:33.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:33.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:33.188 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:25:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:25:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:25:33.606 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2306100103,v1:127.0.0.1:6827/2306100103] [v2:127.0.0.1:6828/2306100103,v1:127.0.0.1:6829/2306100103] exists,up 5af29edd-1ab3-4a77-8e37-d19e41affdd2 2024-09-17T16:25:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:25:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:25:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:25:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:25:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-17T16:25:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:25:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:25:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-17T16:25:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:25:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-17T16:25:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:25:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:25:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:25:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:25:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-17T16:25:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-17T16:25:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:25:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:25:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:25:33.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:25:33.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:25:33.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:25:33.612 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:25:33.612 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:25:33.612 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:25:33.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:25:33.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:25:33.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:25:33.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:25:33.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:25:33.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:25:33.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:25:33.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:25:33.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:25:33.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:25:33.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:25:33.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:25:33.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:25:33.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-17T16:25:33.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:25:33.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=864ef814-4d77-4c90-90e8-d9253561dde6 2024-09-17T16:25:33.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 864ef814-4d77-4c90-90e8-d9253561dde6' 2024-09-17T16:25:33.614 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 864ef814-4d77-4c90-90e8-d9253561dde6 2024-09-17T16:25:33.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:25:33.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB9relmCFltJRAADL77hjPT2OgGfevS9npFng== 2024-09-17T16:25:33.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB9relmCFltJRAADL77hjPT2OgGfevS9npFng=="}' 2024-09-17T16:25:33.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 864ef814-4d77-4c90-90e8-d9253561dde6 -i td/test-erasure-eio/4/new.json 2024-09-17T16:25:34.132 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:25:34.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-17T16:25:34.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9relmCFltJRAADL77hjPT2OgGfevS9npFng== --osd-uuid 864ef814-4d77-4c90-90e8-d9253561dde6 2024-09-17T16:25:34.173 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:25:34.171+0000 7f51dd428640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:25:34.175 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:25:34.173+0000 7f51dd428640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:25:34.176 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:25:34.174+0000 7f51dd428640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:25:34.176 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:25:34.174+0000 7f51dd428640 -1 bdev(0x55999d1dfc00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-17T16:25:34.176 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:25:34.174+0000 7f51dd428640 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-17T16:25:41.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-17T16:25:41.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:25:41.341 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-17T16:25:41.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-17T16:25:41.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:25:41.918 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-17T16:25:41.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-17T16:25:41.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:25:41.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:25:41.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:25:41.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:25:41.945 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:25:41.943+0000 7f744c545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:25:41.966 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:25:41.964+0000 7f744c545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:25:41.967 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:25:41.965+0000 7f744c545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:25:42.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-17T16:25:42.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:25:42.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-17T16:25:42.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:25:42.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:25:42.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:42.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:25:42.389 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:25:42.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:42.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:25:42.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:43.073 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:25:43.071+0000 7f744c545640 -1 Falling back to public interface 2024-09-17T16:25:43.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:43.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:43.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:25:43.832 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:25:43.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:43.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:25:44.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:45.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:45.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:45.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:25:45.232 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:25:45.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:45.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:25:45.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:46.162 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:25:46.160+0000 7f744c545640 -1 osd.4 0 log_to_monitors true 2024-09-17T16:25:46.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:46.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:46.630 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:25:46.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:25:46.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:46.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:25:47.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:48.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:48.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:48.443 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:25:48.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:25:48.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:48.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:25:49.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:50.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:50.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:50.353 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:25:50.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:25:50.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:50.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:25:50.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:51.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:51.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:51.784 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:25:51.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:25:51.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:51.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:25:52.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:53.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:53.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:53.259 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:25:53.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:25:53.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:53.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:25:53.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:54.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:54.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:54.689 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:25:54.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:25:54.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:54.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:25:55.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:56.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:56.134 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:25:56.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:56.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:25:56.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:56.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:25:56.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:57.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:57.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:57.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:25:57.590 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:25:57.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:57.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:25:58.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:25:59.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:25:59.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:25:59.022 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:25:59.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:25:59.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:25:59.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:25:59.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:00.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:00.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:00.440 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:26:00.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:26:00.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:00.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:26:01.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:02.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:02.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:02.018 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:26:02.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:26:02.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:02.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:26:02.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:03.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:03.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:03.693 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:26:03.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:26:03.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:03.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:26:04.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:05.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:05.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:05.566 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:26:05.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:26:05.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:05.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:26:06.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:07.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:07.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:07.274 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:26:07.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:26:07.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:07.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:26:07.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:08.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:08.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:08.959 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:26:08.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:26:08.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:08.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:26:09.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:10.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:10.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:10.979 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:26:10.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:26:10.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:10.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:26:11.403 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3297088368,v1:127.0.0.1:6835/3297088368] [v2:127.0.0.1:6836/3297088368,v1:127.0.0.1:6837/3297088368] exists,up 864ef814-4d77-4c90-90e8-d9253561dde6 2024-09-17T16:26:11.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:26:11.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:26:11.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:26:11.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:26:11.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-17T16:26:11.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:26:11.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:26:11.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-17T16:26:11.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:26:11.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-17T16:26:11.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:26:11.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:26:11.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:26:11.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:26:11.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-17T16:26:11.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-17T16:26:11.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:26:11.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:26:11.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:26:11.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:26:11.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:26:11.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:26:11.409 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:26:11.409 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:26:11.409 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:26:11.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:26:11.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:26:11.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:26:11.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:26:11.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:26:11.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:26:11.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:26:11.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:26:11.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:26:11.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:26:11.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:26:11.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:26:11.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:26:11.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-17T16:26:11.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:26:11.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1278ce6d-a5c2-4620-9cd4-1078987b63b4 2024-09-17T16:26:11.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 1278ce6d-a5c2-4620-9cd4-1078987b63b4' 2024-09-17T16:26:11.411 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 1278ce6d-a5c2-4620-9cd4-1078987b63b4 2024-09-17T16:26:11.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:26:11.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCjrelmyplAGRAAV+g9NxsmDMp8v/w2/7A+rA== 2024-09-17T16:26:11.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCjrelmyplAGRAAV+g9NxsmDMp8v/w2/7A+rA=="}' 2024-09-17T16:26:11.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1278ce6d-a5c2-4620-9cd4-1078987b63b4 -i td/test-erasure-eio/5/new.json 2024-09-17T16:26:11.927 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:26:11.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-17T16:26:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCjrelmyplAGRAAV+g9NxsmDMp8v/w2/7A+rA== --osd-uuid 1278ce6d-a5c2-4620-9cd4-1078987b63b4 2024-09-17T16:26:11.969 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:11.966+0000 7fcd0dc0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:26:12.060 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:12.057+0000 7fcd0dc0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:26:12.061 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:12.058+0000 7fcd0dc0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:26:12.062 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:12.059+0000 7fcd0dc0b640 -1 bdev(0x559560e1fc00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2024-09-17T16:26:12.062 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:12.059+0000 7fcd0dc0b640 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-17T16:26:20.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-17T16:26:20.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:26:20.625 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2024-09-17T16:26:20.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-17T16:26:20.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:26:21.160 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2024-09-17T16:26:21.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-17T16:26:21.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:26:21.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:26:21.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:26:21.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:26:21.188 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:21.186+0000 7fcd21302640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:26:21.202 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:21.201+0000 7fcd21302640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:26:21.203 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:21.202+0000 7fcd21302640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:26:21.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-17T16:26:21.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:26:21.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T16:26:21.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:26:21.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:26:21.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:21.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:26:21.627 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:26:21.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:21.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:22.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:22.556 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:22.554+0000 7fcd21302640 -1 Falling back to public interface 2024-09-17T16:26:23.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:23.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:23.061 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:26:23.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:26:23.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:23.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:23.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:24.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:24.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:24.496 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:26:24.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:26:24.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:24.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:24.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:25.723 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:25.721+0000 7fcd21302640 -1 osd.5 0 log_to_monitors true 2024-09-17T16:26:25.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:25.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:25.922 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:26:25.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:26:25.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:25.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:26.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:27.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:27.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:27.491 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:26:27.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:26:27.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:27.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:28.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:29.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:29.012 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:26:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:26:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:29.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:30.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:30.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:30.411 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:26:30.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:26:30.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:30.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:30.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:31.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:31.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:31.926 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:26:31.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:26:31.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:31.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:32.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:33.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:33.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:33.563 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:26:33.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:26:33.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:33.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:34.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:35.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:35.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:35.031 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:26:35.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:26:35.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:35.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:35.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:36.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:36.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:36.559 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:26:36.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:26:36.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:36.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:36.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:37.970 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:26:37.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:26:37.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:37.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:38.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:39.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:39.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:39.413 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:26:39.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:26:39.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:39.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:40.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:41.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:41.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:41.449 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:26:41.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:26:41.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:41.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:43.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:43.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:43.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:26:43.301 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:26:43.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:43.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:44.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:45.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:45.153 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:26:45.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:45.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:26:45.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:45.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:46.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:47.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:47.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:47.252 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:26:47.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:26:47.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:47.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:47.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:48.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:48.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:48.700 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:26:48.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:26:48.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:48.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:49.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:26:50.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:26:50.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:26:50.213 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:26:50.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:26:50.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:26:50.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:26:50.692 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2717597545,v1:127.0.0.1:6843/2717597545] [v2:127.0.0.1:6844/2717597545,v1:127.0.0.1:6845/2717597545] exists,up 1278ce6d-a5c2-4620-9cd4-1078987b63b4 2024-09-17T16:26:50.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:26:50.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:26:50.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:26:50.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:26:50.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-17T16:26:50.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:26:50.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:26:50.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-17T16:26:50.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:26:50.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-17T16:26:50.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:26:50.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:26:50.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:26:50.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:26:50.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-17T16:26:50.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-17T16:26:50.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:26:50.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:26:50.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:26:50.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:26:50.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:26:50.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:26:50.697 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:26:50.697 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:26:50.697 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:26:50.698 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:26:50.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:26:50.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:26:50.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:26:50.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:26:50.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:26:50.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:26:50.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:26:50.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:26:50.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:26:50.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:26:50.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:26:50.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:26:50.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-17T16:26:50.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:26:50.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=35792737-d445-4d55-b8bc-2869fd42d7c1 2024-09-17T16:26:50.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 35792737-d445-4d55-b8bc-2869fd42d7c1' 2024-09-17T16:26:50.700 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 35792737-d445-4d55-b8bc-2869fd42d7c1 2024-09-17T16:26:50.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:26:50.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDKrelmNbdwKhAAm6BKyKjzPFeoz2StcdLPPQ== 2024-09-17T16:26:50.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDKrelmNbdwKhAAm6BKyKjzPFeoz2StcdLPPQ=="}' 2024-09-17T16:26:50.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 35792737-d445-4d55-b8bc-2869fd42d7c1 -i td/test-erasure-eio/6/new.json 2024-09-17T16:26:51.587 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:26:51.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-17T16:26:51.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKrelmNbdwKhAAm6BKyKjzPFeoz2StcdLPPQ== --osd-uuid 35792737-d445-4d55-b8bc-2869fd42d7c1 2024-09-17T16:26:51.629 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:51.627+0000 7f57fe345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:26:51.630 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:51.629+0000 7f57fe345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:26:51.631 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:51.630+0000 7f57fe345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:26:51.632 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:51.630+0000 7f57fe345640 -1 bdev(0x55ac9d20fc00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2024-09-17T16:26:51.632 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:51.630+0000 7f57fe345640 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-17T16:26:59.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-17T16:26:59.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:26:59.044 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2024-09-17T16:26:59.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-17T16:26:59.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:26:59.579 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2024-09-17T16:26:59.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-17T16:26:59.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:26:59.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:26:59.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:26:59.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:26:59.606 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:59.603+0000 7f7fda42e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:26:59.627 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:59.624+0000 7f7fda42e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:26:59.628 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:26:59.625+0000 7f7fda42e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:27:00.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-17T16:27:00.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:27:00.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-17T16:27:00.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:27:00.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:27:00.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:00.009 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:27:00.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:27:00.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:00.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:00.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:00.985 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:27:00.982+0000 7f7fda42e640 -1 Falling back to public interface 2024-09-17T16:27:01.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:01.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:01.397 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:27:01.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:27:01.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:01.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:01.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:02.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:02.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:02.803 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:27:02.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:27:02.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:02.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:03.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:04.000 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:27:03.997+0000 7f7fda42e640 -1 osd.6 0 log_to_monitors true 2024-09-17T16:27:04.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:04.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:04.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:27:04.264 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:27:04.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:04.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:04.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:05.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:05.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:05.868 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:27:05.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:27:05.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:05.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:06.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:07.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:07.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:07.352 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:27:07.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:27:07.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:07.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:07.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:08.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:08.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:08.881 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:27:08.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:27:08.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:08.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:09.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:10.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:10.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:10.304 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:27:10.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:27:10.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:10.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:10.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:11.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:11.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:11.910 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:27:11.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:27:11.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:11.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:12.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:13.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:13.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:13.343 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:27:13.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:27:13.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:13.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:13.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:14.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:14.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:14.849 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:27:14.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:27:14.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:14.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:15.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:16.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:16.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:16.248 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:27:16.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:27:16.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:16.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:16.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:17.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:17.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:17.648 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:27:17.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:27:17.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:17.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:18.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:19.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:19.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:19.378 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:27:19.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:27:19.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:19.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:20.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:21.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:21.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:21.227 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:27:21.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:27:21.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:21.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:21.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:22.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:22.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:22.994 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:27:22.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:27:22.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:22.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:24.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:25.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:25.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:25.245 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:27:25.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:27:25.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:25.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:26.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:27.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:27.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:27.294 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:27:27.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:27:27.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:27.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:28.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:29.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:29.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:29.206 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:27:29.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:27:29.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:29.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:27:29.743 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3793524497,v1:127.0.0.1:6851/3793524497] [v2:127.0.0.1:6852/3793524497,v1:127.0.0.1:6853/3793524497] exists,up 35792737-d445-4d55-b8bc-2869fd42d7c1 2024-09-17T16:27:29.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 7 2024-09-17T16:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-17T16:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/7 2024-09-17T16:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:27:29.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/7' 2024-09-17T16:27:29.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/7/journal' 2024-09-17T16:27:29.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:27:29.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:27:29.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:27:29.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:27:29.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:27:29.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:27:29.748 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:27:29.748 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:27:29.748 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:27:29.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:27:29.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:27:29.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:27:29.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:27:29.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:27:29.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:27:29.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:27:29.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:27:29.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:27:29.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:27:29.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:27:29.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:27:29.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:27:29.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/7 2024-09-17T16:27:29.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:27:29.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=69060255-ce52-49ed-bddc-0df38746ba03 2024-09-17T16:27:29.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 69060255-ce52-49ed-bddc-0df38746ba03' 2024-09-17T16:27:29.750 INFO:tasks.workunit.client.0.smithi162.stdout:add osd7 69060255-ce52-49ed-bddc-0df38746ba03 2024-09-17T16:27:29.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:27:29.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDxrelmCDx3LRAAn6wJOK5hDxvtA2rCeIUX/w== 2024-09-17T16:27:29.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDxrelmCDx3LRAAn6wJOK5hDxvtA2rCeIUX/w=="}' 2024-09-17T16:27:29.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 69060255-ce52-49ed-bddc-0df38746ba03 -i td/test-erasure-eio/7/new.json 2024-09-17T16:27:30.277 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:27:30.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/7/new.json 2024-09-17T16:27:30.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxrelmCDx3LRAAn6wJOK5hDxvtA2rCeIUX/w== --osd-uuid 69060255-ce52-49ed-bddc-0df38746ba03 2024-09-17T16:27:30.318 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:27:30.316+0000 7fc968820640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:27:30.320 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:27:30.318+0000 7fc968820640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:27:30.321 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:27:30.319+0000 7fc968820640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:27:30.322 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:27:30.320+0000 7fc968820640 -1 bdev(0x56265bebbc00 td/test-erasure-eio/7/block) open stat got: (1) Operation not permitted 2024-09-17T16:27:30.322 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:27:30.320+0000 7fc968820640 -1 bluestore(td/test-erasure-eio/7) _read_fsid unparsable uuid 2024-09-17T16:27:36.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/7/keyring 2024-09-17T16:27:36.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:27:36.584 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd7 key to auth repository 2024-09-17T16:27:36.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-17T16:27:36.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:27:37.155 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.7 2024-09-17T16:27:37.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-17T16:27:37.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:27:37.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:27:37.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:27:37.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:27:37.183 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:27:37.181+0000 7f5192e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:27:37.196 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:27:37.195+0000 7f5192e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:27:37.197 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:27:37.196+0000 7f5192e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:27:37.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-17T16:27:37.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:27:37.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-17T16:27:37.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:27:37.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:27:37.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:37.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:27:37.612 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:27:37.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:37.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:27:37.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:38.551 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:27:38.549+0000 7f5192e1d640 -1 Falling back to public interface 2024-09-17T16:27:39.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:39.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:39.001 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:27:39.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:27:39.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:39.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:27:39.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:40.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:40.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:40.385 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:27:40.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:27:40.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:40.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:27:40.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:27:41.475+0000 7f5192e1d640 -1 osd.7 0 log_to_monitors true 2024-09-17T16:27:41.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:41.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:41.834 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:27:41.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:27:41.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:41.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:27:42.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:43.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:43.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:43.356 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:27:43.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:27:43.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:43.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:27:43.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:44.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:44.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:44.879 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:27:44.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:27:44.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:44.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:27:45.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:46.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:46.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:46.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:27:46.381 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:27:46.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:46.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:27:46.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:47.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:47.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:47.781 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:27:47.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:27:47.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:47.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:27:48.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:49.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:49.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:49.361 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:27:49.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:27:49.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:49.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:27:49.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:50.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:50.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:50.758 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:27:50.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:27:50.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:50.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:27:51.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:52.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:52.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:52.289 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:27:52.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:27:52.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:52.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:27:52.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:53.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:53.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:53.721 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:27:53.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:27:53.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:53.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:27:54.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:55.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:55.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:55.169 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:27:55.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:27:55.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:55.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:27:55.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:56.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:56.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:56.849 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:27:56.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:27:56.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:56.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:27:57.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:27:58.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:27:58.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:27:58.938 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:27:58.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:27:58.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:27:58.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:27:59.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:00.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:00.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:00.784 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:28:00.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:28:00.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:00.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:28:01.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:02.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:02.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:02.597 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:28:02.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:28:02.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:02.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:28:03.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:04.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:04.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:04.349 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:28:04.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:28:04.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:04.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:28:05.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:06.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:06.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:06.025 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:28:06.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:28:06.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:06.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:28:06.607 INFO:tasks.workunit.client.0.smithi162.stdout:osd.7 up in weight 1 up_from 70 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/907065207,v1:127.0.0.1:6859/907065207] [v2:127.0.0.1:6860/907065207,v1:127.0.0.1:6861/907065207] exists,up 69060255-ce52-49ed-bddc-0df38746ba03 2024-09-17T16:28:06.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:28:06.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:28:06.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:28:06.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:28:06.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 8 2024-09-17T16:28:06.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:28:06.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:28:06.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2024-09-17T16:28:06.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:28:06.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/8 2024-09-17T16:28:06.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:28:06.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:28:06.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:28:06.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:28:06.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/8' 2024-09-17T16:28:06.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/8/journal' 2024-09-17T16:28:06.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:28:06.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:28:06.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:28:06.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:28:06.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:28:06.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:28:06.613 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:28:06.613 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:28:06.613 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:28:06.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:28:06.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:28:06.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:28:06.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:28:06.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:28:06.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:28:06.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:28:06.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:28:06.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:28:06.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:28:06.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:28:06.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:28:06.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:28:06.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/8 2024-09-17T16:28:06.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:28:06.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b45adfc2-bc02-487d-884a-b160cef0318d 2024-09-17T16:28:06.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 b45adfc2-bc02-487d-884a-b160cef0318d' 2024-09-17T16:28:06.614 INFO:tasks.workunit.client.0.smithi162.stdout:add osd8 b45adfc2-bc02-487d-884a-b160cef0318d 2024-09-17T16:28:06.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:28:06.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAWrulmmXFoJRAA1dgqeo9MDFkLKfSqGX2vpQ== 2024-09-17T16:28:06.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAWrulmmXFoJRAA1dgqeo9MDFkLKfSqGX2vpQ=="}' 2024-09-17T16:28:06.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b45adfc2-bc02-487d-884a-b160cef0318d -i td/test-erasure-eio/8/new.json 2024-09-17T16:28:07.714 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:28:07.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/8/new.json 2024-09-17T16:28:07.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAWrulmmXFoJRAA1dgqeo9MDFkLKfSqGX2vpQ== --osd-uuid b45adfc2-bc02-487d-884a-b160cef0318d 2024-09-17T16:28:07.757 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:28:07.755+0000 7f72e6218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:28:07.776 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:28:07.774+0000 7f72e6218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:28:07.777 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:28:07.775+0000 7f72e6218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:28:07.777 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:28:07.775+0000 7f72e6218640 -1 bdev(0x5587dec2fc00 td/test-erasure-eio/8/block) open stat got: (1) Operation not permitted 2024-09-17T16:28:07.777 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:28:07.775+0000 7f72e6218640 -1 bluestore(td/test-erasure-eio/8) _read_fsid unparsable uuid 2024-09-17T16:28:14.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/8/keyring 2024-09-17T16:28:14.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:28:14.260 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd8 key to auth repository 2024-09-17T16:28:14.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2024-09-17T16:28:14.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:28:14.802 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.8 2024-09-17T16:28:14.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2024-09-17T16:28:14.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:28:14.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:28:14.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:28:14.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:28:14.830 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:28:14.827+0000 7f73e9dcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:28:14.844 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:28:14.842+0000 7f73e9dcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:28:14.845 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:28:14.843+0000 7f73e9dcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:28:15.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2024-09-17T16:28:15.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:28:15.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2024-09-17T16:28:15.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:28:15.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:28:15.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:15.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:28:15.260 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:28:15.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:15.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:15.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:16.208 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:28:16.206+0000 7f73e9dcc640 -1 Falling back to public interface 2024-09-17T16:28:16.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:16.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:16.659 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:28:16.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:28:16.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:16.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:17.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:18.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:18.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:18.090 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:28:18.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:28:18.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:18.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:18.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:19.204 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:28:19.202+0000 7f73e9dcc640 -1 osd.8 0 log_to_monitors true 2024-09-17T16:28:19.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:19.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:19.498 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:28:19.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:28:19.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:19.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:20.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:21.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:21.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:21.126 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:28:21.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:28:21.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:21.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:21.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:22.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:22.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:22.863 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:28:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:28:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:23.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:24.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:24.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:24.541 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:28:24.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:28:24.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:24.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:24.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:25.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:25.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:25.940 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:28:25.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:28:25.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:25.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:26.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:27.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:27.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:27.375 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:28:27.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:28:27.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:27.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:27.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:28.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:28.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:28.776 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:28:28.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:28:28.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:28.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:29.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:30.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:30.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:30.271 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:28:30.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:28:30.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:30.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:30.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:31.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:31.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:31.687 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:28:31.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:28:31.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:31.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:32.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:33.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:33.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:33.102 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:28:33.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:28:33.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:33.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:34.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:35.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:35.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:35.075 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:28:35.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:28:35.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:35.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:36.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:37.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:37.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:37.141 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:28:37.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:28:37.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:37.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:37.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:38.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:38.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:38.899 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:28:38.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:28:38.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:38.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:40.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:41.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:41.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:41.148 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:28:41.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:28:41.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:41.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:41.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:42.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:42.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:42.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:28:42.573 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:28:42.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:42.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:42.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:43.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:43.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:43.971 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:28:43.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:28:43.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:43.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:44.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:28:45.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:28:45.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:28:45.610 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-09-17T16:28:45.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T16:28:45.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:28:45.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T16:28:46.200 INFO:tasks.workunit.client.0.smithi162.stdout:osd.8 up in weight 1 up_from 81 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/357336750,v1:127.0.0.1:6867/357336750] [v2:127.0.0.1:6868/357336750,v1:127.0.0.1:6869/357336750] exists,up b45adfc2-bc02-487d-884a-b160cef0318d 2024-09-17T16:28:46.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:28:46.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:28:46.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:28:46.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-17T16:28:46.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:28:46.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:28:46.201 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:28:46.202 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:28:46.202 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:28:46.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T16:28:46.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-17T16:28:46.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.0.asok log flush 2024-09-17T16:28:46.281 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:28:46.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-17T16:28:46.295 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:23:18.349+0000 7f8274bb9640 0 load: jerasure load: lrc 2024-09-17T16:28:46.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:443: TEST_ec_recovery_multiple_errors: local poolname=pool-jerasure 2024-09-17T16:28:46.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:444: TEST_ec_recovery_multiple_errors: create_erasure_coded_pool pool-jerasure 4 4 2024-09-17T16:28:46.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:28:46.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-17T16:28:46.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=4 2024-09-17T16:28:46.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-17T16:28:46.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=4 2024-09-17T16:28:46.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-17T16:28:46.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=4 m=4 crush-failure-domain=osd 2024-09-17T16:28:47.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-17T16:28:47.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-17T16:28:48.320 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-17T16:28:48.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:28:49.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-17T16:28:49.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:28:49.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:28:49.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:28:49.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:28:49.335 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:28:49.335 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:28:49.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:28:49.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:28:49.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:28:49.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:28:49.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:28:49.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:28:49.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:28:49.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:28:49.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:28:49.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:28:49.903 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:28:49.903 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:28:49.903 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:28:49.903 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:28:49.903 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:28:49.903 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:28:49.903 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:28:49.903 INFO:tasks.workunit.client.0.smithi162.stderr:8' 2024-09-17T16:28:49.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:28:49.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:28:49.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:28:50.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803842 2024-09-17T16:28:50.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803842 2024-09-17T16:28:50.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842' 2024-09-17T16:28:50.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:28:50.094 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:28:50.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=51539607615 2024-09-17T16:28:50.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 51539607615 2024-09-17T16:28:50.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-51539607615' 2024-09-17T16:28:50.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:28:50.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:28:50.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378678 2024-09-17T16:28:50.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378678 2024-09-17T16:28:50.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-51539607615 2-81604378678' 2024-09-17T16:28:50.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:28:50.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:28:50.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084337 2024-09-17T16:28:50.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084337 2024-09-17T16:28:50.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-51539607615 2-81604378678 3-120259084337' 2024-09-17T16:28:50.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:28:50.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:28:50.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691879 2024-09-17T16:28:50.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691879 2024-09-17T16:28:50.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-51539607615 2-81604378678 3-120259084337 4-171798691879' 2024-09-17T16:28:50.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:28:50.705 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:28:50.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332126 2024-09-17T16:28:50.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332126 2024-09-17T16:28:50.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-51539607615 2-81604378678 3-120259084337 4-171798691879 5-219043332126' 2024-09-17T16:28:50.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:28:50.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:28:50.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=261993005079 2024-09-17T16:28:50.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 261993005079 2024-09-17T16:28:50.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-51539607615 2-81604378678 3-120259084337 4-171798691879 5-219043332126 6-261993005079' 2024-09-17T16:28:50.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:28:50.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T16:28:51.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=300647710736 2024-09-17T16:28:51.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 300647710736 2024-09-17T16:28:51.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-51539607615 2-81604378678 3-120259084337 4-171798691879 5-219043332126 6-261993005079 7-300647710736' 2024-09-17T16:28:51.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:28:51.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T16:28:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350982 2024-09-17T16:28:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350982 2024-09-17T16:28:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-51539607615 2-81604378678 3-120259084337 4-171798691879 5-219043332126 6-261993005079 7-300647710736 8-347892350982' 2024-09-17T16:28:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:28:51.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803842 2024-09-17T16:28:51.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:28:51.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:28:51.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803842 2024-09-17T16:28:51.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:28:51.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803842 2024-09-17T16:28:51.161 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803842 2024-09-17T16:28:51.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803842' 2024-09-17T16:28:51.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:28:51.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803842 -lt 25769803842 2024-09-17T16:28:51.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:28:51.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-51539607615 2024-09-17T16:28:51.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:28:51.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:28:51.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-51539607615 2024-09-17T16:28:51.588 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:28:51.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=51539607615 2024-09-17T16:28:51.589 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 51539607615 2024-09-17T16:28:51.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 51539607615' 2024-09-17T16:28:51.589 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:28:52.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 51539607616 -lt 51539607615 2024-09-17T16:28:52.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:28:52.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378678 2024-09-17T16:28:52.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:28:52.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:28:52.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378678 2024-09-17T16:28:52.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:28:52.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378678 2024-09-17T16:28:52.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378678' 2024-09-17T16:28:52.007 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378678 2024-09-17T16:28:52.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:28:52.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378678 -lt 81604378678 2024-09-17T16:28:52.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:28:52.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084337 2024-09-17T16:28:52.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:28:52.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:28:52.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084337 2024-09-17T16:28:52.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:28:52.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084337 2024-09-17T16:28:52.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084337' 2024-09-17T16:28:52.434 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084337 2024-09-17T16:28:52.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:28:52.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084337 -lt 120259084337 2024-09-17T16:28:52.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:28:52.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691879 2024-09-17T16:28:52.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:28:52.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:28:52.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691879 2024-09-17T16:28:52.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:28:52.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691879 2024-09-17T16:28:52.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691879' 2024-09-17T16:28:52.886 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691879 2024-09-17T16:28:52.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:28:53.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691880 -lt 171798691879 2024-09-17T16:28:53.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:28:53.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-219043332126 2024-09-17T16:28:53.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:28:53.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:28:53.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-219043332126 2024-09-17T16:28:53.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:28:53.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332126 2024-09-17T16:28:53.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 219043332126' 2024-09-17T16:28:53.278 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 219043332126 2024-09-17T16:28:53.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:28:53.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332126 -lt 219043332126 2024-09-17T16:28:53.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:28:53.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-261993005079 2024-09-17T16:28:53.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:28:53.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:28:53.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-261993005079 2024-09-17T16:28:53.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:28:53.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=261993005079 2024-09-17T16:28:53.690 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 261993005079 2024-09-17T16:28:53.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 261993005079' 2024-09-17T16:28:53.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:28:54.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 261993005079 -lt 261993005079 2024-09-17T16:28:54.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:28:54.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-300647710736 2024-09-17T16:28:54.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:28:54.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T16:28:54.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-300647710736 2024-09-17T16:28:54.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:28:54.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=300647710736 2024-09-17T16:28:54.116 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 300647710736 2024-09-17T16:28:54.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 300647710736' 2024-09-17T16:28:54.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T16:28:54.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 300647710737 -lt 300647710736 2024-09-17T16:28:54.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:28:54.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-347892350982 2024-09-17T16:28:54.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:28:54.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T16:28:54.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-347892350982 2024-09-17T16:28:54.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:28:54.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350982 2024-09-17T16:28:54.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 347892350982' 2024-09-17T16:28:54.551 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 347892350982 2024-09-17T16:28:54.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T16:28:54.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350982 -lt 347892350982 2024-09-17T16:28:54.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:28:54.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:28:54.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:28:55.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:28:55.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:28:55.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:28:55.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:28:55.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:28:55.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:28:55.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:28:55.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:28:55.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:28:55.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:28:55.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:28:55.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:28:56.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:28:56.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:28:56.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:28:56.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:446: TEST_ec_recovery_multiple_errors: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-17T16:28:56.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:28:56.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:28:56.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-17T16:28:56.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:28:56.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:28:56.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:28:56.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:28:56.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:28:56.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:28:56.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:28:56.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:28:56.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-17T16:28:56.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:447: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-17T16:28:56.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T16:28:56.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T16:28:56.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T16:28:56.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T16:28:56.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T16:28:56.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T16:28:56.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-17T16:28:56.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T16:28:56.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T16:28:56.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T16:28:56.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-17T16:28:56.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T16:28:56.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-17T16:28:56.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:28:56.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-17T16:28:56.615 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-17T16:28:56.615 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:28:57.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:28:57.006 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:28:57.006 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:28:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:28:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:28:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:28:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:28:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2024-09-17T16:28:57.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-17T16:28:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-17T16:28:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T16:28:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-17T16:28:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T16:28:57.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-17T16:28:57.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T16:28:57.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-17T16:28:57.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T16:28:57.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-17T16:28:57.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T16:28:57.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T16:28:57.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T16:28:57.009 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-17T16:28:57.009 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-17T16:28:57.009 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-17T16:28:57.010 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:28:57.010 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:28:57.010 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:28:57.010 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.3.asok 2024-09-17T16:28:57.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-17T16:28:57.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T16:28:57.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T16:28:57.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T16:28:57.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-17T16:28:57.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-17T16:28:57.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-17T16:28:57.095 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:28:57.095 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:28:57.095 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:28:57.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.3.asok 2024-09-17T16:28:57.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T16:28:57.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-17T16:28:57.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:450: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 3 2024-09-17T16:28:57.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T16:28:57.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T16:28:57.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T16:28:57.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T16:28:57.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T16:28:57.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T16:28:57.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-17T16:28:57.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T16:28:57.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T16:28:57.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T16:28:57.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=3 2024-09-17T16:28:57.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T16:28:57.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-17T16:28:57.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:28:57.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-17T16:28:57.185 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-17T16:28:57.186 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:28:57.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:28:57.621 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:28:57.621 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:28:57.621 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:28:57.621 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:28:57.621 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:28:57.621 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:28:57.621 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2024-09-17T16:28:57.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-17T16:28:57.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-17T16:28:57.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T16:28:57.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-17T16:28:57.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T16:28:57.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/0/type 2024-09-17T16:28:57.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T16:28:57.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-17T16:28:57.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T16:28:57.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-17T16:28:57.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T16:28:57.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T16:28:57.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T16:28:57.624 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-17T16:28:57.624 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:28:57.624 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:28:57.624 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:28:57.624 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:28:57.624 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:28:57.624 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T16:28:57.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-17T16:28:57.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T16:28:57.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T16:28:57.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T16:28:57.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-17T16:28:57.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:28:57.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:28:57.710 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:28:57.710 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:28:57.710 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:28:57.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T16:28:57.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T16:28:57.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.0.asok injectdataerr pool-jerasure myobject 3 2024-09-17T16:28:57.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:451: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 4 2024-09-17T16:28:57.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T16:28:57.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T16:28:57.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T16:28:57.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T16:28:57.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T16:28:57.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T16:28:57.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-17T16:28:57.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T16:28:57.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T16:28:57.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T16:28:57.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=4 2024-09-17T16:28:57.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T16:28:57.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-17T16:28:57.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:28:57.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-17T16:28:57.800 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-17T16:28:57.800 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:28:58.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:28:58.194 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:28:58.194 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:28:58.194 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:28:58.194 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:28:58.194 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:28:58.194 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:28:58.194 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2024-09-17T16:28:58.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-17T16:28:58.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-17T16:28:58.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T16:28:58.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=6 2024-09-17T16:28:58.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T16:28:58.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/6/type 2024-09-17T16:28:58.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T16:28:58.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 6 bluestore_debug_inject_read_err true 2024-09-17T16:28:58.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T16:28:58.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=6 2024-09-17T16:28:58.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T16:28:58.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T16:28:58.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T16:28:58.197 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.6 2024-09-17T16:28:58.197 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.6 2024-09-17T16:28:58.197 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.6 ']' 2024-09-17T16:28:58.197 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:28:58.197 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:28:58.197 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:28:58.197 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.6.asok 2024-09-17T16:28:58.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.6.asok config set bluestore_debug_inject_read_err true 2024-09-17T16:28:58.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T16:28:58.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T16:28:58.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T16:28:58.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.6 2024-09-17T16:28:58.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.6 2024-09-17T16:28:58.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.6 ']' 2024-09-17T16:28:58.284 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:28:58.284 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:28:58.284 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:28:58.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.6.asok 2024-09-17T16:28:58.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T16:28:58.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.6.asok injectdataerr pool-jerasure myobject 4 2024-09-17T16:28:58.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_errors: get_osds pool-jerasure myobject 2024-09-17T16:28:58.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:28:58.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-17T16:28:58.373 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-17T16:28:58.373 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:28:58.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:28:58.780 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:28:58.780 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:28:58.780 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:28:58.780 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:28:58.780 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:28:58.780 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:28:58.780 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2024-09-17T16:28:58.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-17T16:28:58.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_errors: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-17T16:28:58.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_errors: local -a initial_osds 2024-09-17T16:28:58.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:454: TEST_ec_recovery_multiple_errors: local last_osd=7 2024-09-17T16:28:58.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:456: TEST_ec_recovery_multiple_errors: kill_daemons td/test-erasure-eio TERM osd.7 2024-09-17T16:28:58.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:28:58.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:28:58.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:28:58.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:28:58.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:28:59.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:28:59.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:457: TEST_ec_recovery_multiple_errors: ceph osd down 7 2024-09-17T16:29:00.136 INFO:tasks.workunit.client.0.smithi162.stderr:osd.7 is already down. 2024-09-17T16:29:00.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:458: TEST_ec_recovery_multiple_errors: ceph osd out 7 2024-09-17T16:29:00.723 INFO:tasks.workunit.client.0.smithi162.stderr:osd.7 is already out. 2024-09-17T16:29:00.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:461: TEST_ec_recovery_multiple_errors: wait_for_clean 2024-09-17T16:29:00.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:29:00.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:29:00.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:29:00.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:29:00.736 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:29:00.736 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:29:00.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:29:00.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:29:00.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:29:00.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:29:00.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:29:00.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:29:00.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:29:00.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:29:00.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:29:01.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:29:01.276 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:29:01.276 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:29:01.276 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:29:01.276 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:29:01.276 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:29:01.276 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-09-17T16:29:01.277 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-09-17T16:29:01.277 INFO:tasks.workunit.client.0.smithi162.stderr:8' 2024-09-17T16:29:01.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:29:01.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:29:01.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:29:01.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803846 2024-09-17T16:29:01.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803846 2024-09-17T16:29:01.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846' 2024-09-17T16:29:01.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:29:01.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:29:01.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=51539607620 2024-09-17T16:29:01.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 51539607620 2024-09-17T16:29:01.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846 1-51539607620' 2024-09-17T16:29:01.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:29:01.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:29:01.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378683 2024-09-17T16:29:01.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378683 2024-09-17T16:29:01.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846 1-51539607620 2-81604378683' 2024-09-17T16:29:01.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:29:01.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:29:02.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084341 2024-09-17T16:29:02.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084341 2024-09-17T16:29:02.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846 1-51539607620 2-81604378683 3-120259084341' 2024-09-17T16:29:02.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:29:02.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:29:02.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691884 2024-09-17T16:29:02.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691884 2024-09-17T16:29:02.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846 1-51539607620 2-81604378683 3-120259084341 4-171798691884' 2024-09-17T16:29:02.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:29:02.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:29:02.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332130 2024-09-17T16:29:02.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332130 2024-09-17T16:29:02.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846 1-51539607620 2-81604378683 3-120259084341 4-171798691884 5-219043332130' 2024-09-17T16:29:02.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:29:02.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:29:02.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=261993005083 2024-09-17T16:29:02.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 261993005083 2024-09-17T16:29:02.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846 1-51539607620 2-81604378683 3-120259084341 4-171798691884 5-219043332130 6-261993005083' 2024-09-17T16:29:02.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:29:02.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T16:29:02.786 INFO:tasks.workunit.client.0.smithi162.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:29:02.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-17T16:29:02.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-17T16:29:02.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-17T16:29:02.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:29:02.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T16:29:02.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350986 2024-09-17T16:29:02.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350986 2024-09-17T16:29:02.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846 1-51539607620 2-81604378683 3-120259084341 4-171798691884 5-219043332130 6-261993005083 8-347892350986' 2024-09-17T16:29:02.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:29:02.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803846 2024-09-17T16:29:02.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:29:02.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:29:02.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803846 2024-09-17T16:29:02.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:29:02.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803846 2024-09-17T16:29:02.956 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803846 2024-09-17T16:29:02.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803846' 2024-09-17T16:29:02.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:29:03.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803846 -lt 25769803846 2024-09-17T16:29:03.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:29:03.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-51539607620 2024-09-17T16:29:03.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:29:03.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:29:03.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-51539607620 2024-09-17T16:29:03.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:29:03.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=51539607620 2024-09-17T16:29:03.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 51539607620' 2024-09-17T16:29:03.406 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 51539607620 2024-09-17T16:29:03.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:29:03.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 51539607620 -lt 51539607620 2024-09-17T16:29:03.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:29:03.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378683 2024-09-17T16:29:03.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:29:03.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:29:03.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378683 2024-09-17T16:29:03.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:29:03.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378683 2024-09-17T16:29:03.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378683' 2024-09-17T16:29:03.799 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378683 2024-09-17T16:29:03.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:29:04.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378683 -lt 81604378683 2024-09-17T16:29:04.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:29:04.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084341 2024-09-17T16:29:04.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:29:04.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:29:04.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084341 2024-09-17T16:29:04.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:29:04.213 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084341 2024-09-17T16:29:04.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084341 2024-09-17T16:29:04.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084341' 2024-09-17T16:29:04.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:29:04.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084341 -lt 120259084341 2024-09-17T16:29:04.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:29:04.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691884 2024-09-17T16:29:04.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:29:04.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:29:04.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691884 2024-09-17T16:29:04.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:29:04.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691884 2024-09-17T16:29:04.668 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691884 2024-09-17T16:29:04.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691884' 2024-09-17T16:29:04.668 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:29:05.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691884 -lt 171798691884 2024-09-17T16:29:05.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:29:05.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-219043332130 2024-09-17T16:29:05.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:29:05.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:29:05.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-219043332130 2024-09-17T16:29:05.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:29:05.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332130 2024-09-17T16:29:05.062 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 219043332130 2024-09-17T16:29:05.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 219043332130' 2024-09-17T16:29:05.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:29:05.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332130 -lt 219043332130 2024-09-17T16:29:05.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:29:05.485 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-261993005083 2024-09-17T16:29:05.485 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:29:05.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:29:05.487 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-261993005083 2024-09-17T16:29:05.487 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:29:05.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=261993005083 2024-09-17T16:29:05.488 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 261993005083 2024-09-17T16:29:05.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 261993005083' 2024-09-17T16:29:05.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:29:05.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 261993005084 -lt 261993005083 2024-09-17T16:29:05.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:29:05.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-347892350986 2024-09-17T16:29:05.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:29:05.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T16:29:05.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-347892350986 2024-09-17T16:29:05.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:29:05.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350986 2024-09-17T16:29:05.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 347892350986' 2024-09-17T16:29:05.941 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 347892350986 2024-09-17T16:29:05.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T16:29:06.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350986 -lt 347892350986 2024-09-17T16:29:06.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:29:06.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:29:06.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:29:06.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:29:06.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:29:06.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:29:06.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:29:06.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:29:06.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:29:06.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:29:06.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:29:07.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:29:07.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:29:07.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:29:07.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:29:07.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:29:07.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:29:07.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:29:07.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:463: TEST_ec_recovery_multiple_errors: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-17T16:29:07.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:29:07.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:29:07.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-17T16:29:07.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:29:07.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:29:07.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-17T16:29:07.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:29:07.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:29:07.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:465: TEST_ec_recovery_multiple_errors: delete_erasure_coded_pool pool-jerasure 2024-09-17T16:29:07.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:29:07.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-17T16:29:08.322 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-17T16:29:08.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-17T16:29:08.875 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-17T16:29:08.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-17T16:29:08.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:29:08.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:29:08.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:29:08.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:29:08.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:29:08.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:29:08.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:29:08.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:29:09.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:29:09.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:29:09.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:29:09.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:29:09.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:29:09.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:29:09.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:29:09.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:29:09.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:29:09.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:29:09.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:29:09.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:29:09.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:29:09.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:29:09.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:29:09.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:29:09.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:29:09.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:29:09.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:29:09.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:29:09.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-17T16:29:09.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-17T16:29:09.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-17T16:29:09.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-17T16:29:09.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:29:09.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:29:09.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:29:09.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:29:09.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:29:09.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:29:09.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:29:09.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:29:09.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:29:09.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:29:09.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:29:09.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:29:09.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:29:09.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:29:09.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:29:09.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:29:09.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:29:09.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:29:09.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:29:09.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:29:09.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:29:09.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:29:09.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:29:09.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:29:09.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:29:09.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:29:09.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:29:09.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:29:09.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-17T16:29:09.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:29:09.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:29:09.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:29:09.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.87940 2024-09-17T16:29:09.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:29:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T16:29:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:29:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-17T16:29:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-17T16:29:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-17T16:29:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:29:09.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:29:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:29:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-17T16:29:09.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-17T16:29:09.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:29:09.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:29:09.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:29:09.528 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:29:09.528 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:29:09.528 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:29:09.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:29:09.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T16:29:09.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:29:09.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:29:09.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:29:09.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:29:09.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:29:09.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:29:09.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:29:09.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:29:09.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:29:09.739 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:29:09.739 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:29:09.739 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:29:09.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:29:09.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:29:09.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get fsid 2024-09-17T16:29:09.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:29:09.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:29:09.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:29:09.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:29:09.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:29:09.821 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:29:09.821 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:29:09.821 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:29:09.821 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:29:09.821 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:29:09.821 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:29:09.821 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:29:09.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:29:09.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get mon_host 2024-09-17T16:29:09.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-17T16:29:09.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-17T16:29:09.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:29:09.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:29:09.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:29:09.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-17T16:29:09.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:29:10.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:29:10.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:29:10.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:29:10.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:29:10.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:29:10.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:29:10.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:29:10.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:29:10.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:29:10.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-17T16:29:10.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T16:29:11.077 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T16:29:11.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:29:12.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-17T16:29:12.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:29:12.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:29:12.091 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:29:12.091 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:29:12.091 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:29:12.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:29:12.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-17T16:29:12.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-mon.a.asok log flush 2024-09-17T16:29:12.162 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:29:12.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-17T16:29:12.170 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:29:09.559+0000 7fd825527c80 0 load: jerasure load: lrc 2024-09-17T16:29:12.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects td/test-erasure-eio 2024-09-17T16:29:12.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:471: TEST_ec_recovery_multiple_objects: local dir=td/test-erasure-eio 2024-09-17T16:29:12.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:472: TEST_ec_recovery_multiple_objects: local objname=myobject 2024-09-17T16:29:12.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:474: TEST_ec_recovery_multiple_objects: ORIG_ARGS='--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:29:12.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:475: TEST_ec_recovery_multiple_objects: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:29:12.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:476: TEST_ec_recovery_multiple_objects: setup_osds 7 2024-09-17T16:29:12.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-17T16:29:12.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-17T16:29:12.172 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-17T16:29:12.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-17T16:29:12.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:29:12.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-17T16:29:12.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:29:12.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:29:12.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:29:12.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:29:12.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-17T16:29:12.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:29:12.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:29:12.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:29:12.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:29:12.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-17T16:29:12.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-17T16:29:12.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:29:12.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:29:12.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:29:12.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:29:12.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:29:12.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:29:12.177 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:29:12.178 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:29:12.178 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:29:12.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:29:12.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:29:12.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:29:12.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:29:12.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:29:12.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:29:12.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:29:12.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:29:12.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:29:12.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:29:12.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:29:12.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:29:12.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:29:12.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-17T16:29:12.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:29:12.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=733e282a-d85e-4564-8602-9291f1f58a83 2024-09-17T16:29:12.179 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 733e282a-d85e-4564-8602-9291f1f58a83 2024-09-17T16:29:12.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 733e282a-d85e-4564-8602-9291f1f58a83' 2024-09-17T16:29:12.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:29:12.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBYrulmcOStCxAAJu7XJvl/HO2PsobquZb8qw== 2024-09-17T16:29:12.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBYrulmcOStCxAAJu7XJvl/HO2PsobquZb8qw=="}' 2024-09-17T16:29:12.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 733e282a-d85e-4564-8602-9291f1f58a83 -i td/test-erasure-eio/0/new.json 2024-09-17T16:29:12.470 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:29:12.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-17T16:29:12.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBYrulmcOStCxAAJu7XJvl/HO2PsobquZb8qw== --osd-uuid 733e282a-d85e-4564-8602-9291f1f58a83 2024-09-17T16:29:12.514 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:12.512+0000 7f875882a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:29:12.539 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:12.537+0000 7f875882a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:29:12.540 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:12.538+0000 7f875882a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:29:12.541 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:12.539+0000 7f875882a640 -1 bdev(0x5635f1f39c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:29:12.541 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:12.539+0000 7f875882a640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-17T16:29:18.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-17T16:29:18.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:29:18.505 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:29:18.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:29:18.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:29:19.052 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:29:19.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:29:19.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:29:19.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:29:19.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:29:19.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:29:19.083 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:19.080+0000 7efdcd42f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:29:19.095 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:19.092+0000 7efdcd42f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:29:19.095 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:19.093+0000 7efdcd42f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:29:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:29:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:29:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:29:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:29:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:29:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:29:19.510 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:29:19.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:19.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:19.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:20.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:20.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:20.928 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:29:20.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:29:20.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:20.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:20.956 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:20.953+0000 7efdcd42f640 -1 Falling back to public interface 2024-09-17T16:29:21.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:22.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:22.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:22.362 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:29:22.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:29:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:22.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:23.606 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:23.603+0000 7efdcd42f640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:29:23.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:23.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:23.796 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:29:23.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:29:23.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:23.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:24.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:25.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:25.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:25.340 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:29:25.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:29:25.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:25.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:25.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:26.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:26.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:26.743 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:29:26.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:29:26.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:26.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:27.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:28.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:28.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:28.147 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:29:28.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:29:28.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:28.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:28.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:29.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:29.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:29.703 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:29:29.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:29:29.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:29.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:30.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:31.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:31.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:31.132 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:29:31.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:29:31.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:31.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:31.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:32.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:32.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:32.555 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:29:32.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:29:32.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:32.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:32.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:33.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:33.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:33.949 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:29:33.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:29:33.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:33.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:34.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:35.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:35.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:35.473 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:29:35.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:29:35.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:35.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:36.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:37.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:37.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:37.202 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:29:37.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:29:37.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:37.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:38.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:39.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:39.193 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:29:39.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:39.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:29:39.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:39.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:39.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:40.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:40.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:40.997 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:29:40.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:29:40.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:40.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:41.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:42.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:42.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:42.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:29:42.792 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:29:42.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:42.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:43.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:44.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:44.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:44.233 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:29:44.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:29:44.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:44.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:45.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:46.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:46.038 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:29:46.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:46.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:29:46.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:46.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:29:46.657 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2542764948,v1:127.0.0.1:6803/2542764948] [v2:127.0.0.1:6804/2542764948,v1:127.0.0.1:6805/2542764948] exists,up 733e282a-d85e-4564-8602-9291f1f58a83 2024-09-17T16:29:46.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:29:46.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:29:46.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:29:46.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:29:46.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-17T16:29:46.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:29:46.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:29:46.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T16:29:46.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:29:46.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:29:46.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:29:46.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:29:46.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:29:46.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:29:46.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:29:46.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:29:46.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:29:46.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:29:46.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:29:46.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:29:46.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:29:46.663 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:29:46.663 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:29:46.663 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:29:46.663 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:29:46.663 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:29:46.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:29:46.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:29:46.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:29:46.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:29:46.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:29:46.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:29:46.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:29:46.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:29:46.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:29:46.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:29:46.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:29:46.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:29:46.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:29:46.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:29:46.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f377a8d4-71f4-4df2-8184-52494891afc8 2024-09-17T16:29:46.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f377a8d4-71f4-4df2-8184-52494891afc8' 2024-09-17T16:29:46.664 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 f377a8d4-71f4-4df2-8184-52494891afc8 2024-09-17T16:29:46.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:29:46.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB6rulmKAOUKBAAMY8OQansFarCPnNrXdVvnQ== 2024-09-17T16:29:46.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB6rulmKAOUKBAAMY8OQansFarCPnNrXdVvnQ=="}' 2024-09-17T16:29:46.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f377a8d4-71f4-4df2-8184-52494891afc8 -i td/test-erasure-eio/1/new.json 2024-09-17T16:29:47.452 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:29:47.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-17T16:29:47.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6rulmKAOUKBAAMY8OQansFarCPnNrXdVvnQ== --osd-uuid f377a8d4-71f4-4df2-8184-52494891afc8 2024-09-17T16:29:47.497 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:47.495+0000 7f0b6a81e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:29:47.499 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:47.497+0000 7f0b6a81e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:29:47.500 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:47.498+0000 7f0b6a81e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:29:47.500 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:47.498+0000 7f0b6a81e640 -1 bdev(0x55922e113c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-17T16:29:47.500 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:47.498+0000 7f0b6a81e640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-17T16:29:53.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-17T16:29:53.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:29:53.943 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T16:29:53.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T16:29:53.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:29:54.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T16:29:54.485 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T16:29:54.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:29:54.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:29:54.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:29:54.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:29:54.516 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:54.514+0000 7fba799bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:29:54.525 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:54.523+0000 7fba799bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:29:54.527 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:54.524+0000 7fba799bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:29:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T16:29:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:29:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:29:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:29:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:29:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:54.903 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:29:54.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:29:54.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:54.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:29:55.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:56.131 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:56.129+0000 7fba799bf640 -1 Falling back to public interface 2024-09-17T16:29:56.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:56.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:56.326 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:29:56.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:29:56.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:56.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:29:56.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:57.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:57.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:57.761 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:29:57.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:29:57.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:57.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:29:58.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:29:58.831 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:29:58.829+0000 7fba799bf640 -1 osd.1 0 log_to_monitors true 2024-09-17T16:29:59.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:29:59.229 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:29:59.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:29:59.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:29:59.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:29:59.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:29:59.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:00.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:00.772 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:30:00.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:00.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:30:00.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:00.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:30:01.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:02.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:02.274 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:30:02.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:02.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:30:02.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:02.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:30:02.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:03.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:03.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:03.782 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:30:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:30:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:30:04.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:05.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:05.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:05.239 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:30:05.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:30:05.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:05.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:30:05.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:06.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:06.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:06.665 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:30:06.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:30:06.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:06.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:30:07.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:08.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:08.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:08.076 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:30:08.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:30:08.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:08.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:30:08.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:09.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:09.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:09.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:30:09.498 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:30:09.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:09.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:30:10.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:11.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:11.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:11.124 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:30:11.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:30:11.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:11.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:30:11.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:12.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:12.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:12.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:30:12.682 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:30:12.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:12.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:30:13.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:14.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:14.358 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:30:14.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:14.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:30:14.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:14.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:30:15.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:16.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:16.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:16.084 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:30:16.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:30:16.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:16.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:30:16.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:17.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:17.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:17.830 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:30:17.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:30:17.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:17.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:30:18.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:19.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:19.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:19.380 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:30:19.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:30:19.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:19.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:30:19.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:20.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:20.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:20.948 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:30:20.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:30:20.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:20.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:30:21.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:22.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:22.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:22.535 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:30:22.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:30:22.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:22.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:30:23.118 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2079534552,v1:127.0.0.1:6811/2079534552] [v2:127.0.0.1:6812/2079534552,v1:127.0.0.1:6813/2079534552] exists,up f377a8d4-71f4-4df2-8184-52494891afc8 2024-09-17T16:30:23.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:30:23.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:30:23.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:30:23.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:30:23.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-17T16:30:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:30:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:30:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T16:30:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:30:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:30:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:30:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:30:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:30:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:30:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:30:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:30:23.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:30:23.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:30:23.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:30:23.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:30:23.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:30:23.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:30:23.124 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:30:23.124 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:30:23.124 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:30:23.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:30:23.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:30:23.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:30:23.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:30:23.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:30:23.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:30:23.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:30:23.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:30:23.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:30:23.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:30:23.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:30:23.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:30:23.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:30:23.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:30:23.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:30:23.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9e6aad14-c388-425f-a2dc-243e4de7a433 2024-09-17T16:30:23.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 9e6aad14-c388-425f-a2dc-243e4de7a433' 2024-09-17T16:30:23.125 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 9e6aad14-c388-425f-a2dc-243e4de7a433 2024-09-17T16:30:23.126 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:30:23.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCfrulmFBNyCBAAPY+7jDTTMOx5pBGZOEq35g== 2024-09-17T16:30:23.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCfrulmFBNyCBAAPY+7jDTTMOx5pBGZOEq35g=="}' 2024-09-17T16:30:23.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9e6aad14-c388-425f-a2dc-243e4de7a433 -i td/test-erasure-eio/2/new.json 2024-09-17T16:30:23.986 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:30:23.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-17T16:30:23.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCfrulmFBNyCBAAPY+7jDTTMOx5pBGZOEq35g== --osd-uuid 9e6aad14-c388-425f-a2dc-243e4de7a433 2024-09-17T16:30:24.033 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:30:24.031+0000 7f700d3a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:30:24.043 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:30:24.041+0000 7f700d3a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:30:24.044 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:30:24.042+0000 7f700d3a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:30:24.044 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:30:24.042+0000 7f700d3a2640 -1 bdev(0x561ccd89bc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-17T16:30:24.044 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:30:24.042+0000 7f700d3a2640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-17T16:30:30.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-17T16:30:30.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:30:30.192 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T16:30:30.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T16:30:30.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:30:30.741 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T16:30:30.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T16:30:30.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:30:30.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:30:30.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:30:30.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:30:30.772 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:30:30.770+0000 7f947fa2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:30:30.775 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:30:30.773+0000 7f947fa2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:30:30.776 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:30:30.774+0000 7f947fa2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:30:31.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T16:30:31.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:30:31.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:30:31.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:30:31.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:30:31.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:31.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:30:31.191 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:30:31.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:31.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:31.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:32.141 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:30:32.139+0000 7f947fa2e640 -1 Falling back to public interface 2024-09-17T16:30:32.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:32.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:32.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:30:32.618 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:30:32.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:32.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:34.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:34.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:34.045 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:30:34.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:30:34.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:34.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:34.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:34.646 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:30:34.644+0000 7f947fa2e640 -1 osd.2 0 log_to_monitors true 2024-09-17T16:30:35.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:35.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:35.441 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:30:35.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:30:35.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:35.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:35.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:36.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:36.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:36.913 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:30:36.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:30:36.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:36.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:37.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:38.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:38.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:38.346 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:30:38.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:30:38.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:38.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:38.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:39.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:39.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:39.855 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:30:39.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:30:39.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:39.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:40.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:41.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:41.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:41.297 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:30:41.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:30:41.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:41.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:41.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:42.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:42.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:42.738 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:30:42.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:30:42.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:42.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:43.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:44.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:44.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:44.196 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:30:44.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:30:44.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:44.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:44.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:45.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:45.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:45.729 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:30:45.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:30:45.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:45.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:46.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:47.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:47.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:47.225 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:30:47.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:30:47.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:47.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:48.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:49.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:49.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:49.366 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:30:49.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:30:49.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:49.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:50.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:51.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:51.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:51.143 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:30:51.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:30:51.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:51.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:52.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:53.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:53.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:53.384 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:30:53.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:30:53.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:53.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:54.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:55.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:55.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:55.185 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:30:55.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:30:55.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:55.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:55.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:56.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:56.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:56.989 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:30:56.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:30:56.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:56.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:57.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:30:58.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:30:58.850 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:30:58.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:30:58.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:30:58.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:30:58.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:30:59.471 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/159535590,v1:127.0.0.1:6819/159535590] [v2:127.0.0.1:6820/159535590,v1:127.0.0.1:6821/159535590] exists,up 9e6aad14-c388-425f-a2dc-243e4de7a433 2024-09-17T16:30:59.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:30:59.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:30:59.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:30:59.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:30:59.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-17T16:30:59.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:30:59.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:30:59.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T16:30:59.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:30:59.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T16:30:59.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:30:59.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:30:59.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:30:59.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:30:59.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T16:30:59.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T16:30:59.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:30:59.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:30:59.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:30:59.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:30:59.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:30:59.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:30:59.477 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:30:59.477 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:30:59.477 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:30:59.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:30:59.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:30:59.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:30:59.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:30:59.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:30:59.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:30:59.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:30:59.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:30:59.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:30:59.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:30:59.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:30:59.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:30:59.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:30:59.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T16:30:59.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:30:59.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6ab369fb-91b1-43ff-b994-2a436a00aa9e 2024-09-17T16:30:59.478 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 6ab369fb-91b1-43ff-b994-2a436a00aa9e 2024-09-17T16:30:59.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 6ab369fb-91b1-43ff-b994-2a436a00aa9e' 2024-09-17T16:30:59.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:30:59.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDDrulmG9F6HRAAijt3X0d+8Pk8n9V8H8WOlg== 2024-09-17T16:30:59.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDDrulmG9F6HRAAijt3X0d+8Pk8n9V8H8WOlg=="}' 2024-09-17T16:30:59.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6ab369fb-91b1-43ff-b994-2a436a00aa9e -i td/test-erasure-eio/3/new.json 2024-09-17T16:31:00.292 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:31:00.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-17T16:31:00.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDrulmG9F6HRAAijt3X0d+8Pk8n9V8H8WOlg== --osd-uuid 6ab369fb-91b1-43ff-b994-2a436a00aa9e 2024-09-17T16:31:00.337 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:00.334+0000 7f70ff017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:31:00.339 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:00.336+0000 7f70ff017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:31:00.340 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:00.337+0000 7f70ff017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:31:00.341 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:00.338+0000 7f70ff017640 -1 bdev(0x55a0d0417c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-17T16:31:00.341 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:00.338+0000 7f70ff017640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-17T16:31:06.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-17T16:31:06.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:31:06.247 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-17T16:31:06.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T16:31:06.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:31:06.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T16:31:06.788 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T16:31:06.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:31:06.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:31:06.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:31:06.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:31:06.820 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:06.818+0000 7fdfee545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:31:06.831 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:06.829+0000 7fdfee545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:31:06.832 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:06.830+0000 7fdfee545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:31:07.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T16:31:07.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:31:07.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:31:07.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:31:07.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:31:07.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:07.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:31:07.193 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:31:07.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:07.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:07.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:08.164 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:08.162+0000 7fdfee545640 -1 Falling back to public interface 2024-09-17T16:31:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:08.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:08.623 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:31:08.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:31:08.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:08.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:09.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:10.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:10.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:10.031 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:31:10.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:31:10.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:10.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:10.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:11.029 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:11.026+0000 7fdfee545640 -1 osd.3 0 log_to_monitors true 2024-09-17T16:31:11.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:11.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:11.436 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:31:11.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:31:11.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:11.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:11.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:12.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:12.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:12.875 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:31:12.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:31:12.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:12.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:13.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:14.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:14.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:14.341 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:31:14.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:31:14.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:14.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:14.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:15.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:15.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:15.896 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:31:15.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:31:15.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:15.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:16.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:17.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:17.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:17.322 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:31:17.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:31:17.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:17.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:18.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:18.852 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:31:18.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:18.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:31:18.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:18.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:19.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:20.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:20.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:20.341 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:31:20.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:31:20.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:20.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:20.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:21.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:21.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:21.889 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:31:21.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:31:21.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:21.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:22.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:23.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:23.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:23.345 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:31:23.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:31:23.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:23.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:24.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:25.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:25.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:25.060 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:31:25.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:31:25.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:25.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:25.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:26.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:26.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:26.828 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:31:26.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:31:26.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:26.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:27.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:28.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:28.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:28.731 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:31:28.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:31:28.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:28.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:29.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:30.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:30.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:30.577 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:31:30.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:31:30.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:30.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:31.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:32.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:32.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:32.445 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:31:32.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:31:32.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:32.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:34.256 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:31:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:31:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:34.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:35.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:35.918 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:31:35.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:35.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:31:35.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:35.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:31:36.373 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1928829568,v1:127.0.0.1:6827/1928829568] [v2:127.0.0.1:6828/1928829568,v1:127.0.0.1:6829/1928829568] exists,up 6ab369fb-91b1-43ff-b994-2a436a00aa9e 2024-09-17T16:31:36.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:31:36.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:31:36.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:31:36.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:31:36.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-17T16:31:36.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:31:36.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:31:36.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-17T16:31:36.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:31:36.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-17T16:31:36.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:31:36.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:31:36.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:31:36.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:31:36.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-17T16:31:36.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-17T16:31:36.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:31:36.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:31:36.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:31:36.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:31:36.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:31:36.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:31:36.379 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:31:36.379 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:31:36.379 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:31:36.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:31:36.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:31:36.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:31:36.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:31:36.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:31:36.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:31:36.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:31:36.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:31:36.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:31:36.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:31:36.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:31:36.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:31:36.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:31:36.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-17T16:31:36.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:31:36.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f781b8a8-ed97-43d7-aa74-bc2e83808f81 2024-09-17T16:31:36.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 f781b8a8-ed97-43d7-aa74-bc2e83808f81' 2024-09-17T16:31:36.380 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 f781b8a8-ed97-43d7-aa74-bc2e83808f81 2024-09-17T16:31:36.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:31:36.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDorulm4uOnFxAAWVvtFOIp00R3SFrybqnNjA== 2024-09-17T16:31:36.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDorulm4uOnFxAAWVvtFOIp00R3SFrybqnNjA=="}' 2024-09-17T16:31:36.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f781b8a8-ed97-43d7-aa74-bc2e83808f81 -i td/test-erasure-eio/4/new.json 2024-09-17T16:31:37.174 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:31:37.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-17T16:31:37.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDorulm4uOnFxAAWVvtFOIp00R3SFrybqnNjA== --osd-uuid f781b8a8-ed97-43d7-aa74-bc2e83808f81 2024-09-17T16:31:37.221 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:37.217+0000 7f7333011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:31:37.232 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:37.229+0000 7f7333011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:31:37.233 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:37.230+0000 7f7333011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:31:37.234 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:37.231+0000 7f7333011640 -1 bdev(0x563633063c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-17T16:31:37.234 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:37.231+0000 7f7333011640 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-17T16:31:44.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-17T16:31:44.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:31:44.247 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-17T16:31:44.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-17T16:31:44.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:31:44.788 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-17T16:31:44.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-17T16:31:44.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:31:44.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:31:44.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:31:44.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:31:44.820 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:44.817+0000 7fd5ab8fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:31:44.831 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:44.828+0000 7fd5ab8fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:31:44.832 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:44.829+0000 7fd5ab8fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:31:45.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-17T16:31:45.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:31:45.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-17T16:31:45.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:31:45.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:31:45.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:45.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:31:45.273 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:31:45.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:45.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:31:45.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:46.439 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:46.436+0000 7fd5ab8fe640 -1 Falling back to public interface 2024-09-17T16:31:46.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:46.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:46.699 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:31:46.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:31:46.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:46.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:31:47.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:48.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:48.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:48.119 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:31:48.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:31:48.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:48.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:31:48.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:49.087 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:31:49.084+0000 7fd5ab8fe640 -1 osd.4 0 log_to_monitors true 2024-09-17T16:31:49.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:49.518 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:31:49.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:49.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:31:49.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:49.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:31:49.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:50.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:50.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:50.999 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:31:51.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:31:51.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:51.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:31:51.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:52.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:52.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:52.721 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:31:52.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:31:52.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:52.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:31:53.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:54.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:54.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:54.212 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:31:54.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:31:54.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:54.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:31:54.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:55.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:55.625 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:31:55.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:55.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:31:55.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:55.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:31:56.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:57.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:57.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:57.234 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:31:57.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:31:57.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:57.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:31:57.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:31:58.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:31:58.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:31:58.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:31:58.663 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:31:58.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:31:58.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:31:59.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:00.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:00.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:00.061 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:32:00.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:32:00.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:00.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:32:00.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:01.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:01.610 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:32:01.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:01.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:32:01.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:01.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:32:02.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:03.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:03.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:03.101 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:32:03.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:32:03.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:03.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:32:03.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:04.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:04.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:04.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:32:04.714 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:32:04.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:04.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:32:05.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:06.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:06.332 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:32:06.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:06.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:32:06.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:06.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:32:07.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:08.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:08.185 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:32:08.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:08.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:32:08.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:08.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:32:08.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:09.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:09.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:09.863 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:32:09.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:32:09.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:09.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:32:10.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:11.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:11.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:11.609 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:32:11.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:32:11.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:11.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:32:12.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:13.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:13.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:13.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:32:13.350 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:32:13.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:13.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:32:13.751 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3574266123,v1:127.0.0.1:6835/3574266123] [v2:127.0.0.1:6836/3574266123,v1:127.0.0.1:6837/3574266123] exists,up f781b8a8-ed97-43d7-aa74-bc2e83808f81 2024-09-17T16:32:13.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:32:13.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:32:13.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:32:13.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:32:13.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-17T16:32:13.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:32:13.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:32:13.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-17T16:32:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:32:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-17T16:32:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:32:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:32:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:32:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:32:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-17T16:32:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-17T16:32:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:32:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:32:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:32:13.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:32:13.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:32:13.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:32:13.758 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:32:13.758 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:32:13.758 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:32:13.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:32:13.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:32:13.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:32:13.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:32:13.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:32:13.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:32:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:32:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:32:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:32:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:32:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:32:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:32:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:32:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-17T16:32:13.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:32:13.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e82370e3-3829-409c-9458-076eff24de3d 2024-09-17T16:32:13.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 e82370e3-3829-409c-9458-076eff24de3d' 2024-09-17T16:32:13.760 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 e82370e3-3829-409c-9458-076eff24de3d 2024-09-17T16:32:13.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:32:13.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQANr+lmGMYzLhAABfwcOBSiIBV5MOMDSPWINw== 2024-09-17T16:32:13.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQANr+lmGMYzLhAABfwcOBSiIBV5MOMDSPWINw=="}' 2024-09-17T16:32:13.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e82370e3-3829-409c-9458-076eff24de3d -i td/test-erasure-eio/5/new.json 2024-09-17T16:32:14.567 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:32:14.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-17T16:32:14.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANr+lmGMYzLhAABfwcOBSiIBV5MOMDSPWINw== --osd-uuid e82370e3-3829-409c-9458-076eff24de3d 2024-09-17T16:32:14.613 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:32:14.610+0000 7f8d6121e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:32:14.625 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:32:14.622+0000 7f8d6121e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:32:14.626 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:32:14.623+0000 7f8d6121e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:32:14.626 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:32:14.624+0000 7f8d6121e640 -1 bdev(0x5597eba9dc00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2024-09-17T16:32:14.627 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:32:14.624+0000 7f8d6121e640 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-17T16:32:22.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-17T16:32:22.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:32:22.679 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2024-09-17T16:32:22.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-17T16:32:22.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:32:23.237 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2024-09-17T16:32:23.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-17T16:32:23.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:32:23.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:32:23.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:32:23.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:32:23.268 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:32:23.266+0000 7f855fe9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:32:23.270 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:32:23.268+0000 7f855fe9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:32:23.271 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:32:23.269+0000 7f855fe9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:32:23.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-17T16:32:23.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:32:23.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T16:32:23.685 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:32:23.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:32:23.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:32:23.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:23.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:32:23.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:23.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:24.125 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:32:24.123+0000 7f855fe9c640 -1 Falling back to public interface 2024-09-17T16:32:24.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:25.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:25.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:25.139 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:32:25.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:32:25.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:25.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:25.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:26.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:26.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:26.597 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:32:26.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:32:26.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:26.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:27.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:27.107 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:32:27.104+0000 7f855fe9c640 -1 osd.5 0 log_to_monitors true 2024-09-17T16:32:28.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:28.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:28.048 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:32:28.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:32:28.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:28.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:28.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:29.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:29.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:29.842 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:32:29.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:32:29.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:29.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:30.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:31.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:31.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:31.435 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:32:31.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:32:31.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:31.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:31.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:32.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:32.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:32.919 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:32:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:32:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:33.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:34.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:34.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:34.430 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:32:34.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:32:34.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:34.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:34.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:35.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:35.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:35.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:32:35.947 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:32:35.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:35.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:36.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:37.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:37.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:37.509 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:32:37.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:32:37.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:37.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:37.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:38.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:38.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:38.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:32:38.946 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:32:38.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:38.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:39.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:40.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:40.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:40.417 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:32:40.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:32:40.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:40.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:41.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:42.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:42.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:42.453 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:32:42.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:32:42.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:42.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:43.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:44.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:44.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:44.141 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:32:44.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:32:44.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:44.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:45.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:46.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:46.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:46.050 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:32:46.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:32:46.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:46.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:46.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:47.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:47.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:47.845 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:32:47.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:32:47.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:47.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:48.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:49.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:49.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:49.708 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:32:49.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:32:49.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:49.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:50.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:51.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:51.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:51.363 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:32:51.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:32:51.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:51.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:51.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:32:52.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:32:52.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:32:52.848 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:32:52.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:32:52.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:32:52.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:32:53.487 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 51 up_thru 53 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2887358163,v1:127.0.0.1:6843/2887358163] [v2:127.0.0.1:6844/2887358163,v1:127.0.0.1:6845/2887358163] exists,up e82370e3-3829-409c-9458-076eff24de3d 2024-09-17T16:32:53.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:32:53.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:32:53.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:32:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:32:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-17T16:32:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:32:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:32:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-17T16:32:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:32:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-17T16:32:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:32:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:32:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:32:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:32:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-17T16:32:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-17T16:32:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:32:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:32:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:32:53.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:32:53.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:32:53.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:32:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:32:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:32:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:32:53.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:32:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:32:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:32:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:32:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:32:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:32:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:32:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:32:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:32:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:32:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:32:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:32:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:32:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-17T16:32:53.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:32:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a9923b19-3cba-45bf-bf2e-d7f0d5578527 2024-09-17T16:32:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 a9923b19-3cba-45bf-bf2e-d7f0d5578527' 2024-09-17T16:32:53.494 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 a9923b19-3cba-45bf-bf2e-d7f0d5578527 2024-09-17T16:32:53.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:32:53.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA1r+lmiLxwHhAAL+fWt9pZ86h4d8qwN4i8jA== 2024-09-17T16:32:53.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA1r+lmiLxwHhAAL+fWt9pZ86h4d8qwN4i8jA=="}' 2024-09-17T16:32:53.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a9923b19-3cba-45bf-bf2e-d7f0d5578527 -i td/test-erasure-eio/6/new.json 2024-09-17T16:32:54.007 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:32:54.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-17T16:32:54.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1r+lmiLxwHhAAL+fWt9pZ86h4d8qwN4i8jA== --osd-uuid a9923b19-3cba-45bf-bf2e-d7f0d5578527 2024-09-17T16:32:54.054 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:32:54.052+0000 7f02c71bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:32:54.056 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:32:54.054+0000 7f02c71bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:32:54.058 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:32:54.055+0000 7f02c71bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:32:54.058 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:32:54.056+0000 7f02c71bd640 -1 bdev(0x559223eac800 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2024-09-17T16:32:54.058 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:32:54.056+0000 7f02c71bd640 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-17T16:33:00.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-17T16:33:00.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:33:00.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-17T16:33:00.243 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2024-09-17T16:33:00.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:33:00.782 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2024-09-17T16:33:00.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-17T16:33:00.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:33:00.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:33:00.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:33:00.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:33:00.814 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:33:00.812+0000 7f0cb7c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:33:00.847 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:33:00.845+0000 7f0cb7c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:33:00.848 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:33:00.846+0000 7f0cb7c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:33:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-17T16:33:01.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:33:01.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-17T16:33:01.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:33:01.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:33:01.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:01.204 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:33:01.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:33:01.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:01.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:01.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:02.186 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:33:02.184+0000 7f0cb7c0b640 -1 Falling back to public interface 2024-09-17T16:33:02.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:02.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:02.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:33:02.593 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:33:02.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:02.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:02.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:03.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:03.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:03.996 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:33:03.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:33:03.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:03.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:04.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:05.061 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:33:05.059+0000 7f0cb7c0b640 -1 osd.6 0 log_to_monitors true 2024-09-17T16:33:05.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:05.502 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:33:05.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:05.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:33:05.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:05.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:05.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:06.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:06.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:06.901 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:33:06.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:33:06.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:06.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:07.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:08.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:08.776 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:33:08.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:08.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:33:08.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:08.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:09.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:10.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:10.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:10.300 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:33:10.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:33:10.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:10.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:10.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:11.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:11.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:11.739 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:33:11.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:33:11.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:11.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:12.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:13.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:13.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:13.305 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:33:13.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:33:13.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:13.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:13.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:14.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:14.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:14.708 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:33:14.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:33:14.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:14.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:15.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:16.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:16.237 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:33:16.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:16.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:33:16.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:16.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:16.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:17.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:17.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:17.691 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:33:17.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:33:17.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:17.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:18.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:19.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:19.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:19.300 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:33:19.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:33:19.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:19.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:20.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:21.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:21.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:21.119 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:33:21.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:33:21.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:21.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:22.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:23.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:23.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:23.290 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:33:23.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:33:23.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:23.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:24.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:25.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:25.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:25.265 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:33:25.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:33:25.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:25.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:26.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:27.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:27.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:27.204 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:33:27.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:33:27.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:27.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:27.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:28.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:28.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:28.784 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:33:28.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:33:28.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:28.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:29.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:30.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:33:30.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:30.251 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:33:30.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:33:30.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:30.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:33:30.853 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3762235627,v1:127.0.0.1:6851/3762235627] [v2:127.0.0.1:6852/3762235627,v1:127.0.0.1:6853/3762235627] exists,up a9923b19-3cba-45bf-bf2e-d7f0d5578527 2024-09-17T16:33:30.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:33:30.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:33:30.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:33:30.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-17T16:33:30.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:33:30.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:33:30.855 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:33:30.855 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:33:30.855 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:33:30.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T16:33:30.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-17T16:33:30.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.0.asok log flush 2024-09-17T16:33:30.934 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:33:30.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-17T16:33:30.946 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:29:20.957+0000 7efdcd42f640 0 load: jerasure load: lrc 2024-09-17T16:33:30.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:477: TEST_ec_recovery_multiple_objects: CEPH_ARGS='--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:33:30.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:479: TEST_ec_recovery_multiple_objects: local poolname=pool-jerasure 2024-09-17T16:33:30.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:480: TEST_ec_recovery_multiple_objects: create_erasure_coded_pool pool-jerasure 3 2 2024-09-17T16:33:30.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:33:30.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-17T16:33:30.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-17T16:33:30.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-17T16:33:30.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-17T16:33:30.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-17T16:33:30.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-17T16:33:31.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-17T16:33:31.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-17T16:33:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-17T16:33:32.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:33:33.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-17T16:33:33.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:33:33.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:33:33.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:33:33.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:33:33.725 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:33:33.725 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:33:33.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:33:33.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:33:33.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:33:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:33:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:33:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:33:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:33:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:33:33.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:33:34.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:33:34.273 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:33:34.273 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:33:34.273 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:33:34.274 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:33:34.274 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:33:34.274 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-17T16:33:34.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:33:34.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:33:34.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:33:34.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803831 2024-09-17T16:33:34.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803831 2024-09-17T16:33:34.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831' 2024-09-17T16:33:34.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:33:34.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:33:34.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574894 2024-09-17T16:33:34.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574894 2024-09-17T16:33:34.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574894' 2024-09-17T16:33:34.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:33:34.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:33:34.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378662 2024-09-17T16:33:34.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378662 2024-09-17T16:33:34.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574894 2-81604378662' 2024-09-17T16:33:34.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:33:34.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:33:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084319 2024-09-17T16:33:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084319 2024-09-17T16:33:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574894 2-81604378662 3-120259084319' 2024-09-17T16:33:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:33:34.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:33:34.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691865 2024-09-17T16:33:34.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691865 2024-09-17T16:33:34.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574894 2-81604378662 3-120259084319 4-171798691865' 2024-09-17T16:33:34.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:33:34.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:33:35.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332114 2024-09-17T16:33:35.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332114 2024-09-17T16:33:35.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574894 2-81604378662 3-120259084319 4-171798691865 5-219043332114' 2024-09-17T16:33:35.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:33:35.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:33:35.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=257698037766 2024-09-17T16:33:35.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 257698037766 2024-09-17T16:33:35.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574894 2-81604378662 3-120259084319 4-171798691865 5-219043332114 6-257698037766' 2024-09-17T16:33:35.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:33:35.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803831 2024-09-17T16:33:35.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:33:35.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:33:35.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803831 2024-09-17T16:33:35.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:33:35.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803831 2024-09-17T16:33:35.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803831' 2024-09-17T16:33:35.121 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803831 2024-09-17T16:33:35.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:33:35.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803831 -lt 25769803831 2024-09-17T16:33:35.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:33:35.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574894 2024-09-17T16:33:35.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:33:35.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:33:35.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574894 2024-09-17T16:33:35.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:33:35.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574894 2024-09-17T16:33:35.545 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574894 2024-09-17T16:33:35.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574894' 2024-09-17T16:33:35.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:33:35.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574894 -lt 55834574894 2024-09-17T16:33:35.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:33:35.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378662 2024-09-17T16:33:35.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:33:35.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:33:35.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378662 2024-09-17T16:33:35.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:33:35.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378662 2024-09-17T16:33:35.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378662' 2024-09-17T16:33:35.988 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378662 2024-09-17T16:33:35.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:33:36.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378662 -lt 81604378662 2024-09-17T16:33:36.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:33:36.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084319 2024-09-17T16:33:36.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:33:36.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:33:36.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084319 2024-09-17T16:33:36.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:33:36.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084319 2024-09-17T16:33:36.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084319' 2024-09-17T16:33:36.376 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084319 2024-09-17T16:33:36.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:33:36.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084319 -lt 120259084319 2024-09-17T16:33:36.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:33:36.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691865 2024-09-17T16:33:36.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:33:36.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:33:36.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691865 2024-09-17T16:33:36.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:33:36.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691865 2024-09-17T16:33:36.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691865' 2024-09-17T16:33:36.816 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691865 2024-09-17T16:33:36.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:33:37.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691865 -lt 171798691865 2024-09-17T16:33:37.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:33:37.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-219043332114 2024-09-17T16:33:37.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:33:37.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:33:37.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-219043332114 2024-09-17T16:33:37.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:33:37.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332114 2024-09-17T16:33:37.207 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 219043332114 2024-09-17T16:33:37.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 219043332114' 2024-09-17T16:33:37.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:33:37.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332114 -lt 219043332114 2024-09-17T16:33:37.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:33:37.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-257698037766 2024-09-17T16:33:37.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:33:37.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:33:37.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-257698037766 2024-09-17T16:33:37.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:33:37.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=257698037766 2024-09-17T16:33:37.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 257698037766' 2024-09-17T16:33:37.644 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 257698037766 2024-09-17T16:33:37.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:33:38.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037767 -lt 257698037766 2024-09-17T16:33:38.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:33:38.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:33:38.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:33:38.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:33:38.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:33:38.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:33:38.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:33:38.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:33:38.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:33:38.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:33:38.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:33:38.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:33:38.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:33:38.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:33:38.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:33:39.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:33:39.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:33:39.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:33:39.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:482: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test1 2024-09-17T16:33:39.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:33:39.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:33:39.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-09-17T16:33:39.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:33:39.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:33:39.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:33:39.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:33:39.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:33:39.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:33:39.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:33:39.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:33:39.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-09-17T16:33:39.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:483: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test2 2024-09-17T16:33:39.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:33:39.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:33:39.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-09-17T16:33:39.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:33:39.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:33:39.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:33:39.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:33:39.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:33:39.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:33:39.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:33:39.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:33:39.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-09-17T16:33:39.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:484: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test3 2024-09-17T16:33:39.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:33:39.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:33:39.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-09-17T16:33:39.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:33:39.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:33:39.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:33:39.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:33:39.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:33:39.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:33:39.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:33:39.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:33:39.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-09-17T16:33:40.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:486: TEST_ec_recovery_multiple_objects: ceph osd out 0 2024-09-17T16:33:41.178 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 is already out. 2024-09-17T16:33:41.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:489: TEST_ec_recovery_multiple_objects: wait_for_clean 2024-09-17T16:33:41.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:33:41.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:33:41.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:33:41.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:33:41.192 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:33:41.192 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:33:41.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:33:41.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:33:41.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:33:41.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:33:41.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:33:41.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:33:41.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:33:41.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:33:41.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:33:42.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:33:42.013 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:33:42.013 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:33:42.013 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:33:42.013 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:33:42.013 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:33:42.014 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-17T16:33:42.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:33:42.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:33:42.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:33:42.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803835 2024-09-17T16:33:42.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803835 2024-09-17T16:33:42.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835' 2024-09-17T16:33:42.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:33:42.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:33:42.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574897 2024-09-17T16:33:42.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574897 2024-09-17T16:33:42.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574897' 2024-09-17T16:33:42.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:33:42.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:33:42.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378665 2024-09-17T16:33:42.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378665 2024-09-17T16:33:42.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574897 2-81604378665' 2024-09-17T16:33:42.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:33:42.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:33:42.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084323 2024-09-17T16:33:42.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084323 2024-09-17T16:33:42.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574897 2-81604378665 3-120259084323' 2024-09-17T16:33:42.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:33:42.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:33:42.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691868 2024-09-17T16:33:42.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691868 2024-09-17T16:33:42.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574897 2-81604378665 3-120259084323 4-171798691868' 2024-09-17T16:33:42.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:33:42.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:33:42.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332118 2024-09-17T16:33:42.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332118 2024-09-17T16:33:42.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574897 2-81604378665 3-120259084323 4-171798691868 5-219043332118' 2024-09-17T16:33:42.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:33:42.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:33:42.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=257698037770 2024-09-17T16:33:42.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 257698037770 2024-09-17T16:33:42.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574897 2-81604378665 3-120259084323 4-171798691868 5-219043332118 6-257698037770' 2024-09-17T16:33:42.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:33:42.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803835 2024-09-17T16:33:42.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:33:42.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:33:42.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803835 2024-09-17T16:33:42.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:33:42.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803835 2024-09-17T16:33:42.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803835' 2024-09-17T16:33:42.963 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803835 2024-09-17T16:33:42.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:33:43.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803835 -lt 25769803835 2024-09-17T16:33:43.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:33:43.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574897 2024-09-17T16:33:43.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:33:43.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:33:43.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574897 2024-09-17T16:33:43.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:33:43.391 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574897 2024-09-17T16:33:43.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574897 2024-09-17T16:33:43.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574897' 2024-09-17T16:33:43.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:33:43.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574897 -lt 55834574897 2024-09-17T16:33:43.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:33:43.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378665 2024-09-17T16:33:43.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:33:43.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:33:43.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378665 2024-09-17T16:33:43.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:33:43.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378665 2024-09-17T16:33:43.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378665' 2024-09-17T16:33:43.818 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378665 2024-09-17T16:33:43.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:33:44.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378665 -lt 81604378665 2024-09-17T16:33:44.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:33:44.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084323 2024-09-17T16:33:44.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:33:44.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:33:44.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084323 2024-09-17T16:33:44.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:33:44.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084323 2024-09-17T16:33:44.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084323' 2024-09-17T16:33:44.210 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084323 2024-09-17T16:33:44.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:33:44.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084323 -lt 120259084323 2024-09-17T16:33:44.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:33:44.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691868 2024-09-17T16:33:44.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:33:44.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:33:44.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691868 2024-09-17T16:33:44.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:33:44.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691868 2024-09-17T16:33:44.603 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691868 2024-09-17T16:33:44.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691868' 2024-09-17T16:33:44.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:33:44.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691869 -lt 171798691868 2024-09-17T16:33:44.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:33:44.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-219043332118 2024-09-17T16:33:44.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:33:44.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:33:44.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-219043332118 2024-09-17T16:33:44.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:33:44.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332118 2024-09-17T16:33:44.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 219043332118' 2024-09-17T16:33:44.995 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 219043332118 2024-09-17T16:33:44.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:33:45.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332118 -lt 219043332118 2024-09-17T16:33:45.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:33:45.383 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-257698037770 2024-09-17T16:33:45.383 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:33:45.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:33:45.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-257698037770 2024-09-17T16:33:45.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:33:45.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=257698037770 2024-09-17T16:33:45.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 257698037770' 2024-09-17T16:33:45.386 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 257698037770 2024-09-17T16:33:45.386 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:33:45.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037770 -lt 257698037770 2024-09-17T16:33:45.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:33:45.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:33:45.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:33:46.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:33:46.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:33:46.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:33:46.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:33:46.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:33:46.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:33:46.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:33:46.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:33:46.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:33:46.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:33:46.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:33:46.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:33:47.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:33:47.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:33:47.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:33:47.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:491: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test1 2024-09-17T16:33:47.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:33:47.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:33:47.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-09-17T16:33:47.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:33:47.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:33:47.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-09-17T16:33:47.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:33:47.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:33:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:492: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test2 2024-09-17T16:33:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:33:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:33:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-09-17T16:33:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:33:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:33:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-09-17T16:33:47.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:33:47.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:33:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:493: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test3 2024-09-17T16:33:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:33:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:33:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-09-17T16:33:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:33:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:33:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-09-17T16:33:47.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:33:47.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:33:47.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:495: TEST_ec_recovery_multiple_objects: delete_erasure_coded_pool pool-jerasure 2024-09-17T16:33:47.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:33:47.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-17T16:33:47.905 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-17T16:33:47.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-17T16:33:48.404 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-17T16:33:48.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-17T16:33:48.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:33:48.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:33:48.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:33:48.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:33:48.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:33:48.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:33:48.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:33:48.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:33:48.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:33:48.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:33:48.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:33:48.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:33:48.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:33:48.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:33:48.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:33:48.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:33:48.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:33:48.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:33:48.530 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:33:48.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:33:48.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:33:48.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:33:48.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:33:48.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:33:48.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:33:48.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:33:48.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:33:48.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:33:48.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-17T16:33:48.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-17T16:33:48.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-17T16:33:48.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-17T16:33:48.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:33:48.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:33:48.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:33:48.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:33:48.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:33:48.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:33:48.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:33:48.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:33:48.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:33:48.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:33:48.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:33:48.571 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:33:48.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:33:48.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:33:48.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:33:48.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:33:48.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:33:48.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:33:48.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:33:48.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:33:48.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:33:48.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:33:48.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:33:48.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:33:48.577 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:33:48.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:33:48.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:33:48.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:33:48.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-17T16:33:48.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:33:48.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:33:48.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:33:48.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.87940 2024-09-17T16:33:48.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:33:48.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T16:33:48.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:33:48.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-17T16:33:48.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-17T16:33:48.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-17T16:33:48.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:33:48.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:33:48.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:33:48.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-17T16:33:48.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-17T16:33:49.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:33:49.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:33:49.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:33:49.024 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:33:49.024 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:33:49.024 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:33:49.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:33:49.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T16:33:49.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:33:49.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:33:49.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:33:49.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:33:49.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:33:49.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:33:49.209 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:33:49.209 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:33:49.209 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:33:49.210 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:33:49.210 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:33:49.210 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:33:49.210 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:33:49.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:33:49.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get fsid 2024-09-17T16:33:49.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:33:49.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:33:49.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:33:49.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:33:49.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:33:49.291 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:33:49.291 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:33:49.291 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:33:49.291 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:33:49.292 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:33:49.292 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:33:49.292 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:33:49.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:33:49.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get mon_host 2024-09-17T16:33:49.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-17T16:33:49.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-17T16:33:49.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:33:49.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:33:49.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:33:49.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-17T16:33:49.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:33:50.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:33:50.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:33:50.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:33:50.222 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:33:50.222 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:33:50.222 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:33:50.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:33:50.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:33:50.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:33:50.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-17T16:33:50.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T16:33:50.532 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T16:33:50.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:33:51.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-17T16:33:51.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:33:51.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:33:51.545 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:33:51.545 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:33:51.545 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:33:51.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:33:51.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-17T16:33:51.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-mon.a.asok log flush 2024-09-17T16:33:51.616 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:33:51.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-17T16:33:51.625 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:33:49.056+0000 7f2168c62c80 0 load: jerasure load: lrc 2024-09-17T16:33:51.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects_eio td/test-erasure-eio 2024-09-17T16:33:51.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:500: TEST_ec_recovery_multiple_objects_eio: local dir=td/test-erasure-eio 2024-09-17T16:33:51.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:501: TEST_ec_recovery_multiple_objects_eio: local objname=myobject 2024-09-17T16:33:51.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:503: TEST_ec_recovery_multiple_objects_eio: ORIG_ARGS='--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:33:51.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:504: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:33:51.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:505: TEST_ec_recovery_multiple_objects_eio: setup_osds 7 2024-09-17T16:33:51.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-17T16:33:51.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-17T16:33:51.627 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-17T16:33:51.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-17T16:33:51.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:33:51.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-17T16:33:51.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:33:51.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:33:51.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:33:51.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:33:51.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-17T16:33:51.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:33:51.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:33:51.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:33:51.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:33:51.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-17T16:33:51.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-17T16:33:51.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:33:51.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:33:51.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:33:51.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:33:51.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:33:51.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:33:51.632 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:33:51.632 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:33:51.632 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:33:51.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:33:51.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:33:51.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:33:51.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:33:51.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:33:51.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:33:51.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:33:51.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:33:51.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:33:51.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:33:51.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:33:51.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:33:51.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:33:51.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-17T16:33:51.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:33:51.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b09b4e23-2678-4981-a2bd-008d3253387d 2024-09-17T16:33:51.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b09b4e23-2678-4981-a2bd-008d3253387d' 2024-09-17T16:33:51.634 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 b09b4e23-2678-4981-a2bd-008d3253387d 2024-09-17T16:33:51.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:33:51.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBvr+lmiPLGJhAAMg33KuqtMDkwkVHSomRb9w== 2024-09-17T16:33:51.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBvr+lmiPLGJhAAMg33KuqtMDkwkVHSomRb9w=="}' 2024-09-17T16:33:51.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b09b4e23-2678-4981-a2bd-008d3253387d -i td/test-erasure-eio/0/new.json 2024-09-17T16:33:51.917 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:33:51.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-17T16:33:51.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvr+lmiPLGJhAAMg33KuqtMDkwkVHSomRb9w== --osd-uuid b09b4e23-2678-4981-a2bd-008d3253387d 2024-09-17T16:33:51.961 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:33:51.959+0000 7f0096c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:33:52.010 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:33:52.008+0000 7f0096c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:33:52.011 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:33:52.009+0000 7f0096c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:33:52.012 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:33:52.010+0000 7f0096c0d640 -1 bdev(0x5564425ebc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:33:52.012 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:33:52.010+0000 7f0096c0d640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-17T16:33:57.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-17T16:33:57.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:33:57.923 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:33:57.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:33:57.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:33:58.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:33:58.469 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:33:58.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:33:58.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:33:58.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:33:58.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:33:58.500 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:33:58.498+0000 7f44f8415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:33:58.522 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:33:58.520+0000 7f44f8415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:33:58.523 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:33:58.521+0000 7f44f8415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:33:58.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:33:58.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:33:58.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:33:58.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:33:58.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:33:58.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:33:58.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:33:58.937 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:33:58.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:33:58.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:33:59.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:33:59.625 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:33:59.623+0000 7f44f8415640 -1 Falling back to public interface 2024-09-17T16:34:00.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:00.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:00.364 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:34:00.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:34:00.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:00.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:00.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:01.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:01.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:01.798 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:34:01.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:34:01.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:01.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:02.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:02.674 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:34:02.671+0000 7f44f8415640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:34:03.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:03.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:03.195 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:34:03.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:34:03.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:03.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:03.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:04.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:04.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:04.639 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:34:04.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:34:04.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:04.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:05.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:06.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:06.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:06.068 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:34:06.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:34:06.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:06.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:06.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:07.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:07.550 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:34:07.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:07.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:34:07.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:07.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:07.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:08.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:08.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:08.970 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:34:08.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:34:08.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:08.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:09.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:10.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:10.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:10.435 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:34:10.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:34:10.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:10.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:10.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:11.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:11.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:11.860 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:34:11.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:34:11.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:11.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:12.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:13.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:13.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:13.268 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:34:13.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:34:13.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:13.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:13.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:14.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:14.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:14.749 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:34:14.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:34:14.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:14.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:15.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:16.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:16.211 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:34:16.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:16.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:34:16.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:16.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:17.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:17.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:17.799 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:34:17.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:34:17.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:17.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:18.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:19.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:19.717 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:34:19.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:19.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:34:19.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:19.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:20.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:21.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:21.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:21.460 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:34:21.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:34:21.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:21.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:22.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:23.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:23.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:23.586 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:34:23.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:34:23.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:23.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:24.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:25.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:25.040 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:34:25.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:25.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:34:25.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:25.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:25.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:26.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:26.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:26.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:34:26.570 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:34:26.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:26.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:34:26.981 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1075266597,v1:127.0.0.1:6803/1075266597] [v2:127.0.0.1:6804/1075266597,v1:127.0.0.1:6805/1075266597] exists,up b09b4e23-2678-4981-a2bd-008d3253387d 2024-09-17T16:34:26.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:34:26.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:34:26.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:34:26.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:34:26.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-17T16:34:26.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:34:26.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:34:26.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T16:34:26.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:34:26.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:34:26.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:34:26.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:34:26.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:34:26.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:34:26.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:34:26.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:34:26.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:34:26.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:34:26.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:34:26.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:34:26.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:34:26.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:34:26.987 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:34:26.987 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:34:26.987 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:34:26.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:34:26.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:34:26.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:34:26.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:34:26.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:34:26.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:34:26.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:34:26.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:34:26.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:34:26.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:34:26.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:34:26.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:34:26.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:34:26.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:34:26.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:34:26.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8c28e326-85f2-428d-8c57-c2d0fc668b7f 2024-09-17T16:34:26.988 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 8c28e326-85f2-428d-8c57-c2d0fc668b7f 2024-09-17T16:34:26.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 8c28e326-85f2-428d-8c57-c2d0fc668b7f' 2024-09-17T16:34:26.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:34:27.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCTr+lm+M9AABAA3On4kKmidWsEKsfwiuUUbQ== 2024-09-17T16:34:27.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCTr+lm+M9AABAA3On4kKmidWsEKsfwiuUUbQ=="}' 2024-09-17T16:34:27.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8c28e326-85f2-428d-8c57-c2d0fc668b7f -i td/test-erasure-eio/1/new.json 2024-09-17T16:34:27.470 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:34:27.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-17T16:34:27.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTr+lm+M9AABAA3On4kKmidWsEKsfwiuUUbQ== --osd-uuid 8c28e326-85f2-428d-8c57-c2d0fc668b7f 2024-09-17T16:34:27.512 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:34:27.509+0000 7f321ca13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:34:27.516 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:34:27.514+0000 7f321ca13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:34:27.517 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:34:27.515+0000 7f321ca13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:34:27.518 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:34:27.516+0000 7f321ca13640 -1 bdev(0x55dcb8d59c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-17T16:34:27.518 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:34:27.516+0000 7f321ca13640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-17T16:34:33.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-17T16:34:33.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:34:33.294 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T16:34:33.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T16:34:33.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:34:33.842 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T16:34:33.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T16:34:33.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:34:33.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:34:33.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:34:33.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:34:33.874 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:34:33.871+0000 7f38d702a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:34:33.875 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:34:33.874+0000 7f38d702a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:34:33.876 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:34:33.875+0000 7f38d702a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:34:34.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T16:34:34.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:34:34.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:34:34.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:34:34.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:34:34.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:34.291 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:34:34.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:34:34.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:34.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:34.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:35.231 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:34:35.229+0000 7f38d702a640 -1 Falling back to public interface 2024-09-17T16:34:35.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:35.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:35.734 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:34:35.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:34:35.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:35.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:36.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:37.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:37.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:37.185 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:34:37.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:34:37.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:37.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:37.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:38.254 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:34:38.252+0000 7f38d702a640 -1 osd.1 0 log_to_monitors true 2024-09-17T16:34:38.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:38.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:38.591 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:34:38.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:34:38.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:38.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:39.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:40.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:40.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:40.067 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:34:40.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:34:40.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:40.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:40.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:41.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:41.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:41.497 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:34:41.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:34:41.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:41.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:42.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:43.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:43.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:43.071 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:34:43.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:34:43.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:43.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:43.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:44.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:44.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:44.473 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:34:44.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:34:44.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:44.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:44.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:45.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:45.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:45.931 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:34:45.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:34:45.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:45.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:46.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:47.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:47.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:47.390 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:34:47.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:34:47.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:47.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:47.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:48.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:48.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:48.817 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:34:48.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:34:48.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:48.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:49.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:50.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:50.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:50.323 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:34:50.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:34:50.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:50.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:51.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:52.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:52.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:52.147 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:34:52.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:34:52.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:52.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:52.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:53.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:53.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:53.984 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:34:53.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:34:53.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:53.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:54.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:55.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:55.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:55.770 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:34:55.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:34:55.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:55.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:56.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:57.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:57.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:57.497 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:34:57.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:34:57.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:57.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:34:58.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:34:59.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:34:59.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:34:59.585 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:34:59.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:34:59.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:34:59.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:35:00.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:01.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:01.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:01.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:35:01.403 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:35:01.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:01.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:35:01.976 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1374347401,v1:127.0.0.1:6811/1374347401] [v2:127.0.0.1:6812/1374347401,v1:127.0.0.1:6813/1374347401] exists,up 8c28e326-85f2-428d-8c57-c2d0fc668b7f 2024-09-17T16:35:01.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:35:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:35:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:35:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:35:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-17T16:35:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:35:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:35:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T16:35:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:35:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:35:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:35:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:35:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:35:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:35:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:35:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:35:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:35:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:35:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:35:01.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:35:01.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:35:01.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:35:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:35:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:35:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:35:01.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:35:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:35:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:35:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:35:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:35:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:35:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:35:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:35:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:35:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:35:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:35:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:35:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:35:01.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:35:01.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:35:01.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8f2b5b68-5545-4fc6-98a9-7f8f86e9c301 2024-09-17T16:35:01.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 8f2b5b68-5545-4fc6-98a9-7f8f86e9c301' 2024-09-17T16:35:01.984 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 8f2b5b68-5545-4fc6-98a9-7f8f86e9c301 2024-09-17T16:35:01.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:35:02.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC1r+lmwbWWOxAAmU5mXMEEBbJejmFzhxEaLQ== 2024-09-17T16:35:02.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC1r+lmwbWWOxAAmU5mXMEEBbJejmFzhxEaLQ=="}' 2024-09-17T16:35:02.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8f2b5b68-5545-4fc6-98a9-7f8f86e9c301 -i td/test-erasure-eio/2/new.json 2024-09-17T16:35:02.710 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:35:02.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-17T16:35:02.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1r+lmwbWWOxAAmU5mXMEEBbJejmFzhxEaLQ== --osd-uuid 8f2b5b68-5545-4fc6-98a9-7f8f86e9c301 2024-09-17T16:35:02.756 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:02.754+0000 7f5e15e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:35:02.758 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:02.756+0000 7f5e15e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:35:02.759 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:02.757+0000 7f5e15e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:35:02.759 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:02.757+0000 7f5e15e0d640 -1 bdev(0x55a1b7377c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-17T16:35:02.759 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:02.757+0000 7f5e15e0d640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-17T16:35:08.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-17T16:35:08.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:35:08.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T16:35:08.681 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T16:35:08.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:35:09.224 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T16:35:09.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T16:35:09.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:35:09.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:35:09.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:35:09.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:35:09.256 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:09.254+0000 7f3d66e64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:35:09.264 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:09.262+0000 7f3d66e64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:35:09.265 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:09.263+0000 7f3d66e64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:35:09.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T16:35:09.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:35:09.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:35:09.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:35:09.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:35:09.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:09.679 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:35:09.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:35:09.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:09.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:10.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:10.380 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:10.378+0000 7f3d66e64640 -1 Falling back to public interface 2024-09-17T16:35:11.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:11.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:11.068 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:35:11.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:35:11.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:11.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:11.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:12.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:12.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:12.471 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:35:12.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:35:12.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:12.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:12.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:13.211 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:13.209+0000 7f3d66e64640 -1 osd.2 0 log_to_monitors true 2024-09-17T16:35:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:13.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:13.869 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:35:13.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:35:13.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:13.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:14.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:15.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:15.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:15.364 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:35:15.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:35:15.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:15.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:15.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:16.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:16.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:16.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:35:16.782 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:35:16.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:16.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:17.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:18.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:18.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:18.299 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:35:18.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:35:18.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:18.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:18.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:19.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:19.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:19.836 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:35:19.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:35:19.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:19.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:20.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:21.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:21.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:21.384 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:35:21.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:35:21.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:21.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:21.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:22.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:22.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:22.788 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:35:22.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:35:22.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:22.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:23.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:24.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:24.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:24.288 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:35:24.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:35:24.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:24.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:25.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:25.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:25.733 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:35:25.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:35:25.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:25.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:26.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:27.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:27.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:27.733 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:35:27.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:35:27.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:27.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:28.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:29.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:29.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:29.452 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:35:29.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:35:29.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:29.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:30.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:31.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:31.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:31.583 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:35:31.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:35:31.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:31.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:32.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:33.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:33.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:33.284 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:35:33.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:35:33.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:33.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:33.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:34.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:34.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:34.709 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:35:34.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:35:34.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:34.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:35.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:36.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:36.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:36.773 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:35:36.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:35:36.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:36.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:35:37.512 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1741969282,v1:127.0.0.1:6819/1741969282] [v2:127.0.0.1:6820/1741969282,v1:127.0.0.1:6821/1741969282] exists,up 8f2b5b68-5545-4fc6-98a9-7f8f86e9c301 2024-09-17T16:35:37.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:35:37.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:35:37.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:35:37.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:35:37.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-17T16:35:37.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:35:37.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:35:37.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T16:35:37.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:35:37.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T16:35:37.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:35:37.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:35:37.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:35:37.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:35:37.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T16:35:37.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T16:35:37.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:35:37.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:35:37.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:35:37.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:35:37.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:35:37.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:35:37.518 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:35:37.518 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:35:37.518 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:35:37.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:35:37.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:35:37.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:35:37.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:35:37.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:35:37.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:35:37.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:35:37.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:35:37.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:35:37.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:35:37.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:35:37.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:35:37.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:35:37.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T16:35:37.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:35:37.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=907ce5e3-5c68-4087-956f-de5d1193f2da 2024-09-17T16:35:37.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 907ce5e3-5c68-4087-956f-de5d1193f2da' 2024-09-17T16:35:37.520 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 907ce5e3-5c68-4087-956f-de5d1193f2da 2024-09-17T16:35:37.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:35:37.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDZr+lmFyzqHxAAZiogOSItTjkeus0/T/WBBw== 2024-09-17T16:35:37.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDZr+lmFyzqHxAAZiogOSItTjkeus0/T/WBBw=="}' 2024-09-17T16:35:37.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 907ce5e3-5c68-4087-956f-de5d1193f2da -i td/test-erasure-eio/3/new.json 2024-09-17T16:35:38.255 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:35:38.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-17T16:35:38.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZr+lmFyzqHxAAZiogOSItTjkeus0/T/WBBw== --osd-uuid 907ce5e3-5c68-4087-956f-de5d1193f2da 2024-09-17T16:35:38.300 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:38.297+0000 7ff7e9a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:35:38.340 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:38.336+0000 7ff7e9a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:35:38.341 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:38.337+0000 7ff7e9a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:35:38.341 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:38.338+0000 7ff7e9a0f640 -1 bdev(0x5571eb0e9c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-17T16:35:38.341 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:38.338+0000 7ff7e9a0f640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-17T16:35:44.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-17T16:35:44.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:35:44.038 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-17T16:35:44.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T16:35:44.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:35:44.627 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T16:35:44.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T16:35:44.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:35:44.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:35:44.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:35:44.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:35:44.658 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:44.655+0000 7fb6ef7db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:35:44.679 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:44.676+0000 7fb6ef7db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:35:44.681 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:44.678+0000 7fb6ef7db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:35:45.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T16:35:45.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:35:45.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:35:45.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:35:45.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:35:45.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:45.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:35:45.070 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:35:45.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:45.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:35:45.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:46.276 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:46.273+0000 7fb6ef7db640 -1 Falling back to public interface 2024-09-17T16:35:46.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:46.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:46.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:35:46.537 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:35:46.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:46.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:35:46.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:47.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:47.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:47.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:35:47.965 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:35:47.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:47.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:35:48.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:49.025 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:35:49.022+0000 7fb6ef7db640 -1 osd.3 0 log_to_monitors true 2024-09-17T16:35:49.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:49.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:49.431 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:35:49.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:35:49.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:49.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:35:49.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:50.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:50.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:50.937 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:35:50.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:35:50.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:50.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:35:51.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:52.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:52.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:52.618 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:35:52.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:35:52.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:52.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:35:53.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:54.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:54.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:54.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:35:54.080 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:35:54.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:54.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:35:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:55.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:55.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:55.526 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:35:55.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:35:55.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:55.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:35:55.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:56.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:56.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:56.986 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:35:56.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:35:56.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:56.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:35:57.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:35:58.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:35:58.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:35:58.514 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:35:58.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:35:58.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:35:58.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:35:59.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:00.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:00.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:00.035 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:36:00.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:36:00.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:00.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:36:00.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:01.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:01.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:01.518 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:36:01.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:36:01.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:01.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:36:01.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:02.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:02.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:02.943 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:36:02.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:36:02.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:02.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:36:03.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:04.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:04.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:04.750 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:36:04.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:36:04.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:04.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:36:05.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:06.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:06.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:06.737 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:36:06.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:36:06.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:06.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:36:07.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:08.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:08.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:08.776 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:36:08.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:36:08.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:08.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:36:09.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:10.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:10.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:10.579 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:36:10.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:36:10.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:10.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:36:11.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:12.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:12.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:12.680 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:36:12.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:36:12.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:12.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:36:13.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:14.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:14.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:14.127 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:36:14.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:36:14.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:14.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:36:14.579 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3888364038,v1:127.0.0.1:6827/3888364038] [v2:127.0.0.1:6828/3888364038,v1:127.0.0.1:6829/3888364038] exists,up 907ce5e3-5c68-4087-956f-de5d1193f2da 2024-09-17T16:36:14.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:36:14.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:36:14.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:36:14.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:36:14.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-17T16:36:14.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:36:14.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:36:14.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-17T16:36:14.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:36:14.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-17T16:36:14.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:36:14.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:36:14.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:36:14.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:36:14.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-17T16:36:14.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-17T16:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:36:14.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:36:14.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:36:14.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:36:14.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:36:14.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:36:14.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:36:14.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:36:14.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:36:14.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:36:14.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-17T16:36:14.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:36:14.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=290afd6c-3d41-444a-a6dd-ee9e1f95cc2a 2024-09-17T16:36:14.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 290afd6c-3d41-444a-a6dd-ee9e1f95cc2a' 2024-09-17T16:36:14.587 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 290afd6c-3d41-444a-a6dd-ee9e1f95cc2a 2024-09-17T16:36:14.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:36:14.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD+r+lmaY76IxAAheBC7W7AvdxC1TDo3ExyKA== 2024-09-17T16:36:14.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD+r+lmaY76IxAAheBC7W7AvdxC1TDo3ExyKA=="}' 2024-09-17T16:36:14.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 290afd6c-3d41-444a-a6dd-ee9e1f95cc2a -i td/test-erasure-eio/4/new.json 2024-09-17T16:36:15.685 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:36:15.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-17T16:36:15.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+r+lmaY76IxAAheBC7W7AvdxC1TDo3ExyKA== --osd-uuid 290afd6c-3d41-444a-a6dd-ee9e1f95cc2a 2024-09-17T16:36:15.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:36:15.729+0000 7feac6220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:36:15.744 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:36:15.742+0000 7feac6220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:36:15.745 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:36:15.743+0000 7feac6220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:36:15.746 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:36:15.744+0000 7feac6220640 -1 bdev(0x559f15899c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-17T16:36:15.746 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:36:15.744+0000 7feac6220640 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-17T16:36:22.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-17T16:36:22.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:36:22.256 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-17T16:36:22.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-17T16:36:22.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:36:22.823 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-17T16:36:22.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-17T16:36:22.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:36:22.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:36:22.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:36:22.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:36:22.855 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:36:22.853+0000 7f4493411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:36:22.857 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:36:22.855+0000 7f4493411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:36:22.858 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:36:22.856+0000 7f4493411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:36:23.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-17T16:36:23.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:36:23.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-17T16:36:23.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:36:23.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:36:23.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:23.263 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:36:23.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:36:23.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:23.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:23.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:24.196 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:36:24.194+0000 7f4493411640 -1 Falling back to public interface 2024-09-17T16:36:24.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:24.698 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:36:24.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:24.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:36:24.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:24.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:25.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:26.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:26.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:26.134 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:36:26.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:36:26.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:26.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:26.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:27.020 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:36:27.017+0000 7f4493411640 -1 osd.4 0 log_to_monitors true 2024-09-17T16:36:27.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:27.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:27.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:36:27.688 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:36:27.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:27.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:28.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:29.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:29.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:29.276 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:36:29.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:36:29.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:29.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:30.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:31.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:31.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:31.091 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:36:31.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:36:31.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:31.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:31.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:32.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:32.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:32.524 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:36:32.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:36:32.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:32.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:33.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:34.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:34.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:34.100 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:36:34.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:36:34.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:34.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:34.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:35.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:35.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:35.539 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:36:35.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:36:35.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:35.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:36.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:37.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:37.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:37.023 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:36:37.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:36:37.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:37.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:37.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:38.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:38.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:38.444 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:36:38.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:36:38.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:38.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:38.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:39.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:39.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:39.876 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:36:39.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:36:39.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:39.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:40.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:41.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:41.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:41.282 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:36:41.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:36:41.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:41.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:42.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:43.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:43.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:43.212 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:36:43.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:36:43.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:43.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:43.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:44.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:44.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:44.807 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:36:44.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:36:44.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:44.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:45.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:46.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:46.611 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:36:46.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:46.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:36:46.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:46.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:47.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:48.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:48.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:48.161 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:36:48.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:36:48.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:48.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:48.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:49.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:49.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:49.660 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:36:49.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:36:49.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:49.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:50.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:51.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:51.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:51.227 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:36:51.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:36:51.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:51.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:52.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:36:53.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:36:53.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:36:53.032 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-09-17T16:36:53.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T16:36:53.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:36:53.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:36:53.488 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3864965884,v1:127.0.0.1:6835/3864965884] [v2:127.0.0.1:6836/3864965884,v1:127.0.0.1:6837/3864965884] exists,up 290afd6c-3d41-444a-a6dd-ee9e1f95cc2a 2024-09-17T16:36:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:36:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:36:53.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:36:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:36:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-17T16:36:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:36:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:36:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-17T16:36:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:36:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-17T16:36:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:36:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:36:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:36:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:36:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-17T16:36:53.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-17T16:36:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:36:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:36:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:36:53.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:36:53.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:36:53.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:36:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:36:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:36:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:36:53.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:36:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:36:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:36:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:36:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:36:53.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:36:53.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:36:53.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:36:53.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:36:53.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:36:53.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:36:53.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:36:53.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:36:53.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-17T16:36:53.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:36:53.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=29f25cb2-e048-4531-b37b-4829275e0616 2024-09-17T16:36:53.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 29f25cb2-e048-4531-b37b-4829275e0616' 2024-09-17T16:36:53.496 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 29f25cb2-e048-4531-b37b-4829275e0616 2024-09-17T16:36:53.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:36:53.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAlsOlmWvJ2HhAARaByxDrHJzy5eQbvamd5Xw== 2024-09-17T16:36:53.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAlsOlmWvJ2HhAARaByxDrHJzy5eQbvamd5Xw=="}' 2024-09-17T16:36:53.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 29f25cb2-e048-4531-b37b-4829275e0616 -i td/test-erasure-eio/5/new.json 2024-09-17T16:36:54.323 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:36:54.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-17T16:36:54.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAlsOlmWvJ2HhAARaByxDrHJzy5eQbvamd5Xw== --osd-uuid 29f25cb2-e048-4531-b37b-4829275e0616 2024-09-17T16:36:54.369 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:36:54.367+0000 7f018a7dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:36:54.371 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:36:54.369+0000 7f018a7dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:36:54.372 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:36:54.370+0000 7f018a7dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:36:54.372 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:36:54.370+0000 7f018a7dd640 -1 bdev(0x563ef0a67c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2024-09-17T16:36:54.373 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:36:54.370+0000 7f018a7dd640 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-17T16:37:01.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-17T16:37:01.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:37:01.675 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2024-09-17T16:37:01.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-17T16:37:01.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:37:02.226 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2024-09-17T16:37:02.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-17T16:37:02.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:37:02.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:37:02.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:37:02.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:37:02.258 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:37:02.256+0000 7f82b0c30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:37:02.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:37:02.272+0000 7f82b0c30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:37:02.275 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:37:02.273+0000 7f82b0c30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:37:02.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-17T16:37:02.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:37:02.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T16:37:02.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:37:02.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:37:02.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:02.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:37:02.707 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:37:02.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:02.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:03.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:03.640 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:37:03.638+0000 7f82b0c30640 -1 Falling back to public interface 2024-09-17T16:37:04.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:04.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:04.098 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:37:04.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:37:04.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:04.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:04.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:05.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:05.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:05.517 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:37:05.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:37:05.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:05.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:05.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:06.932 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:37:06.930+0000 7f82b0c30640 -1 osd.5 0 log_to_monitors true 2024-09-17T16:37:06.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:06.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:06.953 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:37:06.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:37:06.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:06.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:07.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:08.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:08.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:08.687 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:37:08.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:37:08.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:08.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:09.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:10.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:10.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:10.323 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:37:10.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:37:10.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:10.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:10.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:11.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:11.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:11.938 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:37:11.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:37:11.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:11.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:12.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:13.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:13.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:13.338 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:37:13.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:37:13.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:13.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:13.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:14.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:14.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:14.768 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:37:14.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:37:14.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:14.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:15.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:16.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:16.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:16.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:37:16.190 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:37:16.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:16.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:16.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:17.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:17.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:17.716 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:37:17.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:37:17.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:17.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:18.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:19.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:19.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:19.131 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:37:19.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:37:19.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:19.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:19.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:20.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:20.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:20.545 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:37:20.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:37:20.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:20.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:21.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:22.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:22.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:22.298 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:37:22.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:37:22.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:22.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:23.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:24.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:24.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:24.129 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:37:24.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:37:24.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:24.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:24.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:25.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:25.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:25.941 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:37:25.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:37:25.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:25.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:26.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:27.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:27.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:27.563 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:37:27.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:37:27.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:27.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:28.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:29.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:29.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:29.333 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:37:29.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:37:29.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:29.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:30.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:31.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:31.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:31.290 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:37:31.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:37:31.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:31.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:31.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:32.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:32.721 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-09-17T16:37:32.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:32.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T16:37:32.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:32.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:37:33.112 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 51 up_thru 53 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/919608569,v1:127.0.0.1:6843/919608569] [v2:127.0.0.1:6844/919608569,v1:127.0.0.1:6845/919608569] exists,up 29f25cb2-e048-4531-b37b-4829275e0616 2024-09-17T16:37:33.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:37:33.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:37:33.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:37:33.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:37:33.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-17T16:37:33.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:37:33.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:37:33.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-17T16:37:33.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:37:33.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-17T16:37:33.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:37:33.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:37:33.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:37:33.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:37:33.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-17T16:37:33.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-17T16:37:33.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:37:33.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:37:33.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:37:33.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:37:33.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:37:33.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:37:33.118 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:37:33.118 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:37:33.118 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:37:33.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:37:33.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:37:33.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:37:33.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:37:33.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:37:33.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:37:33.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:37:33.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:37:33.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:37:33.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:37:33.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:37:33.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:37:33.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:37:33.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-17T16:37:33.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:37:33.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a6c3fe5a-47c6-4d7d-ac46-2c2c9d2390c7 2024-09-17T16:37:33.119 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 a6c3fe5a-47c6-4d7d-ac46-2c2c9d2390c7 2024-09-17T16:37:33.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 a6c3fe5a-47c6-4d7d-ac46-2c2c9d2390c7' 2024-09-17T16:37:33.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:37:33.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBNsOlm82YPCBAALJkHekOLKtDdXE2rHbCq7A== 2024-09-17T16:37:33.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBNsOlm82YPCBAALJkHekOLKtDdXE2rHbCq7A=="}' 2024-09-17T16:37:33.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a6c3fe5a-47c6-4d7d-ac46-2c2c9d2390c7 -i td/test-erasure-eio/6/new.json 2024-09-17T16:37:33.733 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:37:33.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-17T16:37:33.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNsOlm82YPCBAALJkHekOLKtDdXE2rHbCq7A== --osd-uuid a6c3fe5a-47c6-4d7d-ac46-2c2c9d2390c7 2024-09-17T16:37:33.778 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:37:33.776+0000 7f058ee9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:37:33.780 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:37:33.778+0000 7f058ee9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:37:33.781 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:37:33.779+0000 7f058ee9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:37:33.782 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:37:33.779+0000 7f058ee9c640 -1 bdev(0x5572811a9c00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2024-09-17T16:37:33.782 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:37:33.780+0000 7f058ee9c640 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-17T16:37:40.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-17T16:37:40.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:37:40.060 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2024-09-17T16:37:40.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-17T16:37:40.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:37:40.646 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2024-09-17T16:37:40.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-17T16:37:40.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:37:40.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:37:40.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:37:40.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:37:40.678 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:37:40.676+0000 7efd88c12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:37:40.694 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:37:40.692+0000 7efd88c12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:37:40.695 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:37:40.693+0000 7efd88c12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:37:41.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-17T16:37:41.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:37:41.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-17T16:37:41.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:37:41.109 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:37:41.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:37:41.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:41.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:37:41.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:41.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:37:41.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:42.042 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:37:42.040+0000 7efd88c12640 -1 Falling back to public interface 2024-09-17T16:37:42.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:42.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:42.544 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:37:42.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:37:42.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:42.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:37:42.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:43.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:43.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:43.972 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:37:43.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:37:43.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:43.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:37:44.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:44.899 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:37:44.897+0000 7efd88c12640 -1 osd.6 0 log_to_monitors true 2024-09-17T16:37:45.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:45.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:45.377 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:37:45.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:37:45.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:45.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:37:45.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:46.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:46.805 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:37:46.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:46.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:37:46.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:46.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:37:47.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:48.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:48.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:48.587 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:37:48.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:37:48.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:48.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:37:49.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:50.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:50.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:50.301 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:37:50.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:37:50.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:50.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:37:50.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:51.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:51.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:51.752 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:37:51.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:37:51.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:51.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:37:52.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:53.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:53.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:53.199 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:37:53.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:37:53.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:53.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:37:53.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:54.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:54.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:54.606 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:37:54.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:37:54.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:54.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:37:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:56.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:56.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:56.150 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:37:56.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:37:56.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:56.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:37:56.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:57.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:57.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:57.556 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:37:57.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:37:57.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:57.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:37:57.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:37:58.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:37:58.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:37:58.967 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:37:58.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:37:58.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:37:58.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:37:59.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:00.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:00.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:00.949 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:38:00.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:38:00.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:00.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:38:01.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:02.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:02.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:02.576 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:38:02.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:38:02.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:02.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:38:03.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:04.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:04.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:04.740 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:38:04.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:38:04.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:04.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:38:05.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:06.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:06.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:06.276 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:38:06.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:38:06.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:06.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:38:06.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:07.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:07.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:07.770 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:38:07.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:38:07.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:07.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:38:08.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:09.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:09.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:09.505 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:38:09.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:38:09.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:09.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:38:09.916 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1150296072,v1:127.0.0.1:6851/1150296072] [v2:127.0.0.1:6852/1150296072,v1:127.0.0.1:6853/1150296072] exists,up a6c3fe5a-47c6-4d7d-ac46-2c2c9d2390c7 2024-09-17T16:38:09.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:38:09.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:38:09.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:38:09.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-17T16:38:09.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:38:09.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:38:09.918 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:38:09.918 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:38:09.918 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:38:09.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T16:38:09.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-17T16:38:09.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.0.asok log flush 2024-09-17T16:38:09.997 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:38:10.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-17T16:38:10.009 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:33:59.627+0000 7f44f8415640 0 load: jerasure load: lrc 2024-09-17T16:38:10.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:506: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS='--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:38:10.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:508: TEST_ec_recovery_multiple_objects_eio: local poolname=pool-jerasure 2024-09-17T16:38:10.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:509: TEST_ec_recovery_multiple_objects_eio: create_erasure_coded_pool pool-jerasure 3 2 2024-09-17T16:38:10.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:38:10.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-17T16:38:10.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-17T16:38:10.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-17T16:38:10.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-17T16:38:10.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-17T16:38:10.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-17T16:38:11.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-17T16:38:11.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-17T16:38:11.879 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-17T16:38:11.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:38:12.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-17T16:38:12.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:38:12.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:38:12.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:38:12.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:38:12.893 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:38:12.893 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:38:12.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:38:12.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:38:12.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:38:12.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:38:12.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:38:12.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:38:12.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:38:12.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:38:12.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:38:13.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:38:13.376 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:38:13.376 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:38:13.376 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:38:13.376 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:38:13.376 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:38:13.376 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-17T16:38:13.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:38:13.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:38:13.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:38:13.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803828 2024-09-17T16:38:13.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803828 2024-09-17T16:38:13.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803828' 2024-09-17T16:38:13.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:38:13.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:38:13.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574893 2024-09-17T16:38:13.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574893 2024-09-17T16:38:13.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803828 1-55834574893' 2024-09-17T16:38:13.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:38:13.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:38:13.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378661 2024-09-17T16:38:13.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378661 2024-09-17T16:38:13.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803828 1-55834574893 2-81604378661' 2024-09-17T16:38:13.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:38:13.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:38:13.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084322 2024-09-17T16:38:13.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084322 2024-09-17T16:38:13.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803828 1-55834574893 2-81604378661 3-120259084322' 2024-09-17T16:38:13.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:38:13.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:38:14.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691867 2024-09-17T16:38:14.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691867 2024-09-17T16:38:14.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803828 1-55834574893 2-81604378661 3-120259084322 4-171798691867' 2024-09-17T16:38:14.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:38:14.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:38:14.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332113 2024-09-17T16:38:14.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332113 2024-09-17T16:38:14.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803828 1-55834574893 2-81604378661 3-120259084322 4-171798691867 5-219043332113' 2024-09-17T16:38:14.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:38:14.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:38:14.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=257698037765 2024-09-17T16:38:14.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 257698037765 2024-09-17T16:38:14.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803828 1-55834574893 2-81604378661 3-120259084322 4-171798691867 5-219043332113 6-257698037765' 2024-09-17T16:38:14.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:38:14.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803828 2024-09-17T16:38:14.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:38:14.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:38:14.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803828 2024-09-17T16:38:14.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:38:14.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803828 2024-09-17T16:38:14.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803828' 2024-09-17T16:38:14.299 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803828 2024-09-17T16:38:14.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:38:14.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803828 -lt 25769803828 2024-09-17T16:38:14.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:38:14.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574893 2024-09-17T16:38:14.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:38:14.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:38:14.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574893 2024-09-17T16:38:14.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:38:14.788 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574893 2024-09-17T16:38:14.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574893 2024-09-17T16:38:14.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574893' 2024-09-17T16:38:14.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:38:15.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574893 -lt 55834574893 2024-09-17T16:38:15.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:38:15.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378661 2024-09-17T16:38:15.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:38:15.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:38:15.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378661 2024-09-17T16:38:15.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:38:15.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378661 2024-09-17T16:38:15.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378661' 2024-09-17T16:38:15.215 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378661 2024-09-17T16:38:15.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:38:15.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378661 -lt 81604378661 2024-09-17T16:38:15.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:38:15.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084322 2024-09-17T16:38:15.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:38:15.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:38:15.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084322 2024-09-17T16:38:15.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:38:15.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084322 2024-09-17T16:38:15.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084322' 2024-09-17T16:38:15.659 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084322 2024-09-17T16:38:15.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:38:16.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084322 -lt 120259084322 2024-09-17T16:38:16.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:38:16.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691867 2024-09-17T16:38:16.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:38:16.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:38:16.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691867 2024-09-17T16:38:16.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:38:16.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691867 2024-09-17T16:38:16.097 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691867 2024-09-17T16:38:16.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691867' 2024-09-17T16:38:16.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:38:16.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691867 -lt 171798691867 2024-09-17T16:38:16.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:38:16.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-219043332113 2024-09-17T16:38:16.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:38:16.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:38:16.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-219043332113 2024-09-17T16:38:16.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:38:16.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332113 2024-09-17T16:38:16.544 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 219043332113 2024-09-17T16:38:16.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 219043332113' 2024-09-17T16:38:16.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:38:16.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332113 -lt 219043332113 2024-09-17T16:38:16.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:38:16.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-257698037765 2024-09-17T16:38:16.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:38:16.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:38:16.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-257698037765 2024-09-17T16:38:16.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:38:16.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=257698037765 2024-09-17T16:38:16.996 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 257698037765 2024-09-17T16:38:16.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 257698037765' 2024-09-17T16:38:16.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:38:17.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037766 -lt 257698037765 2024-09-17T16:38:17.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:38:17.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:38:17.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:38:17.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:38:17.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:38:17.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:38:17.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:38:17.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:38:17.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:38:17.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:38:17.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:38:18.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:38:18.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:38:18.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:38:18.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:38:18.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:38:18.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:38:18.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:38:18.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:511: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test1 2024-09-17T16:38:18.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:38:18.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:38:18.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-09-17T16:38:18.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:38:18.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:38:18.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:38:18.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:38:18.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:38:18.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:38:18.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:38:18.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:38:18.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-09-17T16:38:19.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:512: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test2 2024-09-17T16:38:19.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:38:19.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:38:19.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-09-17T16:38:19.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:38:19.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:38:19.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:38:19.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:38:19.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:38:19.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:38:19.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:38:19.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:38:19.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-09-17T16:38:19.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:513: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test3 2024-09-17T16:38:19.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:38:19.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:38:19.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-09-17T16:38:19.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:38:19.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:38:19.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:38:19.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:38:19.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:38:19.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:38:19.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:38:19.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:38:19.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-09-17T16:38:19.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:516: TEST_ec_recovery_multiple_objects_eio: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-17T16:38:19.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T16:38:19.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T16:38:19.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T16:38:19.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T16:38:19.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T16:38:19.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T16:38:19.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-17T16:38:19.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T16:38:19.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T16:38:19.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T16:38:19.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-17T16:38:19.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T16:38:19.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-17T16:38:19.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:38:19.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-17T16:38:19.722 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-17T16:38:19.722 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:38:20.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:38:20.150 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:38:20.150 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:38:20.150 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:38:20.150 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-17T16:38:20.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2024-09-17T16:38:20.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-09-17T16:38:20.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T16:38:20.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-17T16:38:20.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T16:38:20.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-17T16:38:20.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T16:38:20.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-17T16:38:20.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T16:38:20.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-17T16:38:20.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T16:38:20.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T16:38:20.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T16:38:20.153 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-17T16:38:20.153 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-17T16:38:20.153 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-17T16:38:20.153 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:38:20.153 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:38:20.153 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:38:20.153 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.3.asok 2024-09-17T16:38:20.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-17T16:38:20.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T16:38:20.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T16:38:20.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T16:38:20.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-17T16:38:20.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-17T16:38:20.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-17T16:38:20.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:38:20.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:38:20.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:38:20.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.3.asok 2024-09-17T16:38:20.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T16:38:20.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-17T16:38:20.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:517: TEST_ec_recovery_multiple_objects_eio: ceph osd out 0 2024-09-17T16:38:20.848 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 is already out. 2024-09-17T16:38:20.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:520: TEST_ec_recovery_multiple_objects_eio: wait_for_clean 2024-09-17T16:38:20.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:38:20.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:38:20.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:38:20.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:38:20.861 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:38:20.861 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:38:20.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:38:20.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:38:20.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:38:20.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:38:20.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:38:20.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:38:20.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:38:20.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:38:20.950 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:38:21.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:38:21.556 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:38:21.556 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:38:21.556 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:38:21.556 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:38:21.556 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:38:21.556 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-17T16:38:21.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:38:21.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:38:21.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:38:21.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803832 2024-09-17T16:38:21.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803832 2024-09-17T16:38:21.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803832' 2024-09-17T16:38:21.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:38:21.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:38:21.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574897 2024-09-17T16:38:21.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574897 2024-09-17T16:38:21.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803832 1-55834574897' 2024-09-17T16:38:21.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:38:21.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:38:22.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378664 2024-09-17T16:38:22.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378664 2024-09-17T16:38:22.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803832 1-55834574897 2-81604378664' 2024-09-17T16:38:22.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:38:22.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:38:22.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084325 2024-09-17T16:38:22.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084325 2024-09-17T16:38:22.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803832 1-55834574897 2-81604378664 3-120259084325' 2024-09-17T16:38:22.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:38:22.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:38:22.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691870 2024-09-17T16:38:22.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691870 2024-09-17T16:38:22.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803832 1-55834574897 2-81604378664 3-120259084325 4-171798691870' 2024-09-17T16:38:22.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:38:22.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:38:22.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332117 2024-09-17T16:38:22.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332117 2024-09-17T16:38:22.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803832 1-55834574897 2-81604378664 3-120259084325 4-171798691870 5-219043332117' 2024-09-17T16:38:22.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:38:22.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:38:22.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=257698037769 2024-09-17T16:38:22.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 257698037769 2024-09-17T16:38:22.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803832 1-55834574897 2-81604378664 3-120259084325 4-171798691870 5-219043332117 6-257698037769' 2024-09-17T16:38:22.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:38:22.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803832 2024-09-17T16:38:22.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:38:22.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:38:22.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803832 2024-09-17T16:38:22.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:38:22.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803832 2024-09-17T16:38:22.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803832' 2024-09-17T16:38:22.761 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803832 2024-09-17T16:38:22.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:38:23.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803832 -lt 25769803832 2024-09-17T16:38:23.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:38:23.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574897 2024-09-17T16:38:23.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:38:23.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:38:23.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574897 2024-09-17T16:38:23.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:38:23.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574897 2024-09-17T16:38:23.154 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574897 2024-09-17T16:38:23.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574897' 2024-09-17T16:38:23.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:38:23.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574897 -lt 55834574897 2024-09-17T16:38:23.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:38:23.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378664 2024-09-17T16:38:23.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:38:23.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:38:23.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378664 2024-09-17T16:38:23.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:38:23.625 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378664 2024-09-17T16:38:23.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378664 2024-09-17T16:38:23.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378664' 2024-09-17T16:38:23.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:38:24.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378664 -lt 81604378664 2024-09-17T16:38:24.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:38:24.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084325 2024-09-17T16:38:24.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:38:24.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:38:24.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084325 2024-09-17T16:38:24.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:38:24.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084325 2024-09-17T16:38:24.013 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084325 2024-09-17T16:38:24.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084325' 2024-09-17T16:38:24.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:38:24.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084326 -lt 120259084325 2024-09-17T16:38:24.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:38:24.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691870 2024-09-17T16:38:24.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:38:24.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:38:24.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691870 2024-09-17T16:38:24.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:38:24.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691870 2024-09-17T16:38:24.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691870' 2024-09-17T16:38:24.419 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 171798691870 2024-09-17T16:38:24.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:38:24.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691871 -lt 171798691870 2024-09-17T16:38:24.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:38:24.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-219043332117 2024-09-17T16:38:24.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:38:24.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:38:24.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-219043332117 2024-09-17T16:38:24.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:38:24.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332117 2024-09-17T16:38:24.838 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 219043332117 2024-09-17T16:38:24.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 219043332117' 2024-09-17T16:38:24.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:38:25.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332117 -lt 219043332117 2024-09-17T16:38:25.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:38:25.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-257698037769 2024-09-17T16:38:25.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:38:25.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:38:25.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-257698037769 2024-09-17T16:38:25.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:38:25.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=257698037769 2024-09-17T16:38:25.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 257698037769' 2024-09-17T16:38:25.265 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 257698037769 2024-09-17T16:38:25.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:38:25.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037769 -lt 257698037769 2024-09-17T16:38:25.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:38:25.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:38:25.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:38:26.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:38:26.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:38:26.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:38:26.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:38:26.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:38:26.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:38:26.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:38:26.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:38:26.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T16:38:26.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:38:26.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:38:26.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:38:27.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-17T16:38:27.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-17T16:38:27.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T16:38:27.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-17T16:38:27.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T16:38:27.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T16:38:27.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T16:38:27.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:38:27.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:38:27.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:38:27.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:38:27.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:38:27.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:38:27.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:38:27.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T16:38:27.588 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:38:27.588 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:38:27.588 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:38:28.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-17T16:38:28.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-17T16:38:28.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T16:38:28.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T16:38:28.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T16:38:28.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T16:38:28.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T16:38:28.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T16:38:28.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T16:38:28.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T16:38:28.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T16:38:28.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T16:38:28.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T16:38:28.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T16:38:28.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T16:38:28.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:38:28.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:38:28.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:38:28.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:38:28.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:38:28.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:38:28.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:38:29.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:38:29.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:38:29.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:38:29.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:38:29.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:38:29.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:38:29.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:38:29.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:522: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test1 2024-09-17T16:38:29.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:38:29.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:38:29.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-09-17T16:38:29.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:38:29.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:38:29.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-09-17T16:38:29.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:38:29.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:38:29.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:523: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test2 2024-09-17T16:38:29.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:38:29.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:38:29.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-09-17T16:38:29.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:38:29.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:38:29.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-09-17T16:38:29.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:38:29.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:38:29.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:524: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test3 2024-09-17T16:38:29.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:38:29.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:38:29.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-09-17T16:38:29.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:38:29.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:38:29.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-09-17T16:38:29.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:38:29.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:38:29.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:526: TEST_ec_recovery_multiple_objects_eio: delete_erasure_coded_pool pool-jerasure 2024-09-17T16:38:29.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:38:29.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-17T16:38:30.425 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-17T16:38:30.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-17T16:38:31.148 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-17T16:38:31.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-17T16:38:31.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:38:31.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:38:31.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:38:31.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:38:31.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:38:31.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:38:31.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:38:31.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:38:31.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:38:31.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:38:31.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:38:31.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:38:31.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:38:31.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:38:31.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:38:31.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:38:31.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:38:31.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:38:31.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:38:31.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:38:31.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:38:31.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:38:31.311 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:38:31.311 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:38:31.311 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:38:31.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:38:31.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:38:31.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:38:31.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-17T16:38:31.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-17T16:38:31.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-17T16:38:31.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-17T16:38:31.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:38:31.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:38:31.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:38:31.314 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:38:31.314 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:38:31.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:38:31.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:38:31.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:38:31.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:38:31.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:38:31.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:38:31.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:38:31.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:38:31.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:38:31.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:38:31.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:38:31.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:38:31.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:38:31.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:38:31.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:38:31.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:38:31.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:38:31.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:38:31.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:38:31.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:38:31.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:38:31.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:38:31.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:38:31.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-17T16:38:31.324 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:38:31.324 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:38:31.324 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:38:31.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.87940 2024-09-17T16:38:31.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:38:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T16:38:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:38:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-17T16:38:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-17T16:38:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-17T16:38:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:38:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:38:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:38:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-17T16:38:31.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-17T16:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:38:31.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T16:38:31.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:38:31.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:38:31.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:38:31.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:38:31.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:38:31.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:38:31.993 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:38:31.994 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:38:31.994 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:38:31.994 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:38:31.994 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:38:31.994 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:38:31.994 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:38:31.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:38:31.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get fsid 2024-09-17T16:38:32.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:38:32.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:38:32.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:38:32.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:38:32.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:38:32.080 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:38:32.080 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:38:32.080 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:38:32.081 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:38:32.081 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:38:32.081 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:38:32.081 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:38:32.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:38:32.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get mon_host 2024-09-17T16:38:32.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-17T16:38:32.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-17T16:38:32.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:38:32.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:38:32.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:38:32.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-17T16:38:32.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:38:33.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:38:33.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:38:33.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:38:33.014 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:38:33.014 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:38:33.014 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:38:33.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:38:33.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:38:33.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:38:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-17T16:38:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T16:38:33.324 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T16:38:33.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:38:34.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-17T16:38:34.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:38:34.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:38:34.336 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:38:34.336 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:38:34.336 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:38:34.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:38:34.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-17T16:38:34.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-mon.a.asok log flush 2024-09-17T16:38:34.409 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:38:34.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-17T16:38:34.418 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:38:31.841+0000 7fa9b4429c80 0 load: jerasure load: lrc 2024-09-17T16:38:34.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_unfound td/test-erasure-eio 2024-09-17T16:38:34.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:629: TEST_ec_recovery_unfound: local dir=td/test-erasure-eio 2024-09-17T16:38:34.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:630: TEST_ec_recovery_unfound: local objname=myobject 2024-09-17T16:38:34.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:631: TEST_ec_recovery_unfound: local lastobj=100 2024-09-17T16:38:34.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:633: TEST_ec_recovery_unfound: local testobj=obj75 2024-09-17T16:38:34.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:635: TEST_ec_recovery_unfound: ORIG_ARGS='--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:38:34.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:636: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-17T16:38:34.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-17T16:38:34.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:638: TEST_ec_recovery_unfound: setup_osds 5 2024-09-17T16:38:34.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-09-17T16:38:34.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-17T16:38:34.420 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-09-17T16:38:34.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-09-17T16:38:34.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:38:34.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-17T16:38:34.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:38:34.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:38:34.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:38:34.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:38:34.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-17T16:38:34.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-17T16:38:34.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:38:34.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:38:34.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:38:34.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-17T16:38:34.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-17T16:38:34.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:38:34.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:38:34.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:38:34.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:38:34.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:38:34.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:38:34.424 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:38:34.424 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:38:34.424 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:38:34.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:38:34.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:38:34.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:38:34.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:38:34.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:38:34.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:38:34.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:38:34.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:38:34.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:38:34.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:38:34.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:38:34.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:38:34.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:38:34.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-17T16:38:34.425 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:38:34.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=449fae49-3378-409f-a128-5a1e680bfa79 2024-09-17T16:38:34.426 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 449fae49-3378-409f-a128-5a1e680bfa79 2024-09-17T16:38:34.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 449fae49-3378-409f-a128-5a1e680bfa79' 2024-09-17T16:38:34.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:38:34.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCKsOlmsT2cGhAAfgskVzGde0bTjCv7czcGwQ== 2024-09-17T16:38:34.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCKsOlmsT2cGhAAfgskVzGde0bTjCv7czcGwQ=="}' 2024-09-17T16:38:34.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 449fae49-3378-409f-a128-5a1e680bfa79 -i td/test-erasure-eio/0/new.json 2024-09-17T16:38:34.717 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:38:34.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-17T16:38:34.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCKsOlmsT2cGhAAfgskVzGde0bTjCv7czcGwQ== --osd-uuid 449fae49-3378-409f-a128-5a1e680bfa79 2024-09-17T16:38:34.765 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:38:34.763+0000 7f08a33cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:38:34.786 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:38:34.784+0000 7f08a33cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:38:34.787 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:38:34.785+0000 7f08a33cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:38:34.787 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:38:34.785+0000 7f08a33cc640 -1 bdev(0x56163f69dc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:38:34.787 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:38:34.785+0000 7f08a33cc640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-17T16:38:40.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-17T16:38:40.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:38:40.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:38:40.726 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:38:40.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:38:41.272 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:38:41.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:38:41.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:38:41.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:38:41.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:38:41.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:38:41.309 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:38:41.306+0000 7f4995611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:38:41.310 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:38:41.309+0000 7f4995611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:38:41.311 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:38:41.310+0000 7f4995611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:38:41.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:38:41.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:38:41.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:38:41.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:38:41.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:38:41.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:41.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:38:41.679 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:38:41.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:41.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:38:42.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:42.399 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:38:42.397+0000 7f4995611640 -1 Falling back to public interface 2024-09-17T16:38:43.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:43.117 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:38:43.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:43.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:38:43.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:43.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:38:43.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:44.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:44.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:44.559 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:38:44.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:38:44.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:44.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:38:44.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:45.462 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:38:45.460+0000 7f4995611640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:38:45.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:45.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:45.969 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:38:45.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:38:45.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:45.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:38:46.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:47.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:47.478 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:38:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:38:47.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:47.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:38:47.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:48.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:48.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:48.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:38:48.897 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:38:48.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:48.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:38:49.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:50.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:50.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:50.419 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:38:50.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:38:50.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:50.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:38:50.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:51.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:51.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:51.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:38:51.857 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:38:51.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:51.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:38:52.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:53.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:53.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:53.410 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:38:53.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:38:53.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:53.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:38:53.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:54.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:54.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:54.837 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:38:54.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:38:54.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:54.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:38:55.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:56.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:56.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:56.385 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:38:56.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:38:56.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:56.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:38:56.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:57.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:57.794 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:38:57.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:57.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:38:57.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:57.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:38:58.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:38:59.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:38:59.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:38:59.341 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:38:59.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:38:59.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:38:59.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:39:00.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:01.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:01.415 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:39:01.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:01.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:39:01.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:01.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:39:02.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:03.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:03.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:03.166 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:39:03.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:39:03.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:03.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:39:03.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:04.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:04.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:04.759 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:39:04.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:39:04.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:04.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:39:05.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:06.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:06.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:06.554 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:39:06.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:39:06.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:06.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:39:07.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:08.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:08.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:08.288 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:39:08.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:39:08.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:08.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:39:08.794 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/705601402,v1:127.0.0.1:6803/705601402] [v2:127.0.0.1:6804/705601402,v1:127.0.0.1:6805/705601402] exists,up 449fae49-3378-409f-a128-5a1e680bfa79 2024-09-17T16:39:08.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:39:08.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:39:08.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:39:08.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:39:08.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-17T16:39:08.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:39:08.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:39:08.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T16:39:08.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:39:08.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:39:08.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-17T16:39:08.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:39:08.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:39:08.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:39:08.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:39:08.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:39:08.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:39:08.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:39:08.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:39:08.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:39:08.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:39:08.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:39:08.805 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:39:08.805 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:39:08.805 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:39:08.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:39:08.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:39:08.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:39:08.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:39:08.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:39:08.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:39:08.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:39:08.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:39:08.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:39:08.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:39:08.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:39:08.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:39:08.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:39:08.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:39:08.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:39:08.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0ba68639-6212-48dd-8363-1df18c25056d 2024-09-17T16:39:08.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 0ba68639-6212-48dd-8363-1df18c25056d' 2024-09-17T16:39:08.807 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 0ba68639-6212-48dd-8363-1df18c25056d 2024-09-17T16:39:08.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:39:08.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCssOlm2qXdMBAAScRf3DhqZWii0Ybyc173Fg== 2024-09-17T16:39:08.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCssOlm2qXdMBAAScRf3DhqZWii0Ybyc173Fg=="}' 2024-09-17T16:39:08.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0ba68639-6212-48dd-8363-1df18c25056d -i td/test-erasure-eio/1/new.json 2024-09-17T16:39:09.427 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:39:09.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-17T16:39:09.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCssOlm2qXdMBAAScRf3DhqZWii0Ybyc173Fg== --osd-uuid 0ba68639-6212-48dd-8363-1df18c25056d 2024-09-17T16:39:09.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:09.476+0000 7fdac1e1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:39:09.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:09.483+0000 7fdac1e1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:39:09.486 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:09.484+0000 7fdac1e1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:39:09.487 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:09.485+0000 7fdac1e1a640 -1 bdev(0x5625ed619c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-17T16:39:09.487 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:09.485+0000 7fdac1e1a640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-17T16:39:15.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-17T16:39:15.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:39:15.462 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T16:39:15.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T16:39:15.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:39:16.013 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T16:39:16.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T16:39:16.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:39:16.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:39:16.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:39:16.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:39:16.049 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:16.047+0000 7fe8f5ddd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:39:16.051 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:16.049+0000 7fe8f5ddd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:39:16.052 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:16.050+0000 7fe8f5ddd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:39:16.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T16:39:16.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:39:16.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:39:16.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:39:16.427 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:39:16.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:39:16.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:16.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:39:16.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:16.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:16.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:17.420 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:17.418+0000 7fe8f5ddd640 -1 Falling back to public interface 2024-09-17T16:39:17.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:17.858 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:39:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:39:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:18.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:19.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:19.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:19.264 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:39:19.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:39:19.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:19.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:19.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:20.371 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:20.369+0000 7fe8f5ddd640 -1 osd.1 0 log_to_monitors true 2024-09-17T16:39:20.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:20.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:20.682 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:39:20.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:39:20.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:20.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:21.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:22.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:22.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:22.175 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:39:22.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:39:22.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:22.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:22.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:23.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:23.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:23.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:39:23.682 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:39:23.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:23.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:24.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:25.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:25.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:25.171 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:39:25.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:39:25.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:25.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:25.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:26.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:26.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:26.735 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:39:26.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:39:26.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:26.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:27.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:28.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:28.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:28.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:39:28.215 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:39:28.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:28.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:28.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:29.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:29.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:29.696 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:39:29.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:39:29.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:29.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:30.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:31.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:31.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:31.184 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:39:31.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:39:31.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:31.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:31.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:32.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:32.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:32.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:39:32.616 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:39:32.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:32.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:33.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:34.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:34.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:34.171 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:39:34.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:39:34.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:34.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:34.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:35.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:35.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:35.949 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:39:35.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:39:35.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:35.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:36.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:37.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:37.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:37.818 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:39:37.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:39:37.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:37.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:38.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:39.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:39.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:39.620 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:39:39.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:39:39.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:39.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:40.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:41.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:41.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:41.800 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:39:41.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:39:41.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:41.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:42.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:43.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:43.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:43.318 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:39:43.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:39:43.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:43.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:39:43.853 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1199842619,v1:127.0.0.1:6811/1199842619] [v2:127.0.0.1:6812/1199842619,v1:127.0.0.1:6813/1199842619] exists,up 0ba68639-6212-48dd-8363-1df18c25056d 2024-09-17T16:39:43.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:39:43.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:39:43.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:39:43.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:39:43.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-17T16:39:43.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:39:43.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:39:43.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T16:39:43.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:39:43.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:39:43.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-17T16:39:43.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:39:43.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:39:43.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:39:43.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:39:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:39:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:39:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:39:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:39:43.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:39:43.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:39:43.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:39:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:39:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:39:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:39:43.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:39:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:39:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:39:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:39:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:39:43.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:39:43.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:39:43.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:39:43.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:39:43.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:39:43.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:39:43.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:39:43.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:39:43.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:39:43.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:39:43.859 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 f634ce53-bc95-449b-89fc-41431f8afc45 2024-09-17T16:39:43.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f634ce53-bc95-449b-89fc-41431f8afc45 2024-09-17T16:39:43.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 f634ce53-bc95-449b-89fc-41431f8afc45' 2024-09-17T16:39:43.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:39:43.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDPsOlmCM1rNBAAF8FPyPVUnpSsK74ogZ/XWQ== 2024-09-17T16:39:43.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDPsOlmCM1rNBAAF8FPyPVUnpSsK74ogZ/XWQ=="}' 2024-09-17T16:39:43.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f634ce53-bc95-449b-89fc-41431f8afc45 -i td/test-erasure-eio/2/new.json 2024-09-17T16:39:44.824 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:39:44.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-17T16:39:44.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPsOlmCM1rNBAAF8FPyPVUnpSsK74ogZ/XWQ== --osd-uuid f634ce53-bc95-449b-89fc-41431f8afc45 2024-09-17T16:39:44.873 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:44.870+0000 7f1ed4ae0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:39:44.923 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:44.920+0000 7f1ed4ae0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:39:44.925 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:44.921+0000 7f1ed4ae0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:39:44.925 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:44.921+0000 7f1ed4ae0640 -1 bdev(0x55d7e5279c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-17T16:39:44.925 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:44.922+0000 7f1ed4ae0640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-17T16:39:51.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-17T16:39:51.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:39:51.498 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T16:39:51.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T16:39:51.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:39:52.049 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T16:39:52.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T16:39:52.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:39:52.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:39:52.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:39:52.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:39:52.084 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:52.082+0000 7fe666ca5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:39:52.088 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:52.086+0000 7fe666ca5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:39:52.089 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:52.088+0000 7fe666ca5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:39:52.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T16:39:52.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:39:52.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:39:52.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:39:52.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:39:52.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:52.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:39:52.512 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:39:52.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:52.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:39:52.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:53.689 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:53.687+0000 7fe666ca5640 -1 Falling back to public interface 2024-09-17T16:39:53.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:53.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:53.947 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:39:53.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:39:53.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:53.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:39:54.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:55.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:55.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:55.365 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:39:55.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:39:55.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:55.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:39:55.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:56.652 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:39:56.650+0000 7fe666ca5640 -1 osd.2 0 log_to_monitors true 2024-09-17T16:39:56.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:56.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:56.808 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:39:56.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:39:56.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:56.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:39:57.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:58.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:58.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:58.448 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:39:58.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:39:58.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:58.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:39:58.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:39:59.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:39:59.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:39:59.900 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:39:59.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:39:59.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:39:59.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:40:00.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:01.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:01.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:01.404 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:40:01.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:40:01.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:01.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:40:01.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:02.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:02.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:02.893 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:40:02.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:40:02.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:02.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:40:03.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:04.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:04.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:04.380 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:40:04.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:40:04.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:04.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:40:04.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:05.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:05.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:05.790 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:40:05.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:40:05.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:05.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:40:06.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:07.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:07.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:07.197 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:40:07.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:40:07.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:07.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:40:07.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:08.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:08.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:08.692 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:40:08.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:40:08.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:08.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:40:09.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:10.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:10.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:10.110 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:40:10.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:40:10.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:10.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:40:11.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:12.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:12.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:12.257 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:40:12.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:40:12.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:12.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:40:13.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:14.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:14.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:14.161 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:40:14.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:40:14.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:14.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:40:14.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:15.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:15.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:15.670 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:40:15.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:40:15.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:15.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:40:16.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:17.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:17.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:17.826 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:40:17.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:40:17.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:17.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:40:18.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:19.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:19.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:19.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:40:19.382 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:40:19.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:19.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:40:20.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:21.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:21.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:21.103 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:40:21.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:40:21.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:21.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:40:21.809 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4028603165,v1:127.0.0.1:6819/4028603165] [v2:127.0.0.1:6820/4028603165,v1:127.0.0.1:6821/4028603165] exists,up f634ce53-bc95-449b-89fc-41431f8afc45 2024-09-17T16:40:21.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:40:21.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:40:21.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:40:21.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:40:21.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-17T16:40:21.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:40:21.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:40:21.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T16:40:21.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:40:21.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T16:40:21.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-17T16:40:21.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:40:21.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:40:21.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:40:21.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T16:40:21.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T16:40:21.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:40:21.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:40:21.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:40:21.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:40:21.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:40:21.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:40:21.817 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:40:21.817 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:40:21.817 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:40:21.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:40:21.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:40:21.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:40:21.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:40:21.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:40:21.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:40:21.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:40:21.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:40:21.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:40:21.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:40:21.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:40:21.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:40:21.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:40:21.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T16:40:21.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:40:21.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7f7a3b9a-b289-4432-84a7-7b991ae08cbf 2024-09-17T16:40:21.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 7f7a3b9a-b289-4432-84a7-7b991ae08cbf' 2024-09-17T16:40:21.819 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 7f7a3b9a-b289-4432-84a7-7b991ae08cbf 2024-09-17T16:40:21.819 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:40:21.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD1sOlmeYbbMRAAxkchwjfxFZbUABQNTFcmHw== 2024-09-17T16:40:21.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD1sOlmeYbbMRAAxkchwjfxFZbUABQNTFcmHw=="}' 2024-09-17T16:40:21.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7f7a3b9a-b289-4432-84a7-7b991ae08cbf -i td/test-erasure-eio/3/new.json 2024-09-17T16:40:22.514 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:40:22.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-17T16:40:22.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1sOlmeYbbMRAAxkchwjfxFZbUABQNTFcmHw== --osd-uuid 7f7a3b9a-b289-4432-84a7-7b991ae08cbf 2024-09-17T16:40:22.564 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:40:22.561+0000 7fc55441d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:40:22.629 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:40:22.624+0000 7fc55441d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:40:22.629 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:40:22.625+0000 7fc55441d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:40:22.629 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:40:22.626+0000 7fc55441d640 -1 bdev(0x55b749a0a800 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-17T16:40:22.629 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:40:22.626+0000 7fc55441d640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-17T16:40:28.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-17T16:40:28.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:40:28.574 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-17T16:40:28.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T16:40:28.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:40:29.122 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T16:40:29.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T16:40:29.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:40:29.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:40:29.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:40:29.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:40:29.157 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:40:29.154+0000 7f4e5b9c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:40:29.175 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:40:29.172+0000 7f4e5b9c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:40:29.176 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:40:29.173+0000 7f4e5b9c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:40:29.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T16:40:29.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:40:29.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:40:29.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:40:29.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:40:29.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:29.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:40:29.581 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:40:29.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:29.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:29.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:30.026 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:40:30.023+0000 7f4e5b9c1640 -1 Falling back to public interface 2024-09-17T16:40:30.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:30.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:30.976 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:40:30.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:40:30.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:30.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:31.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:32.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:32.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:32.417 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:40:32.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:40:32.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:32.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:40:32.657+0000 7f4e5b9c1640 -1 osd.3 0 log_to_monitors true 2024-09-17T16:40:32.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:33.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:33.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:33.904 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:40:33.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:40:33.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:33.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:34.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:35.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:35.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:35.688 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:40:35.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:40:35.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:35.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:36.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:37.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:37.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:37.099 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:40:37.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:40:37.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:37.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:37.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:38.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:38.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:38.593 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:40:38.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:40:38.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:38.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:39.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:40.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:40.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:40.016 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:40:40.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:40:40.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:40.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:40.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:41.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:41.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:41.503 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:40:41.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:40:41.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:41.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:41.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:42.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:42.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:42.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:40:42.912 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:40:42.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:42.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:43.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:44.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:44.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:44.369 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:40:44.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:40:44.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:44.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:44.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:45.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:45.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:45.943 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:40:45.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:40:45.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:45.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:46.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:47.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:47.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:47.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:40:47.606 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:40:47.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:47.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:48.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:49.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:49.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:49.079 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:40:49.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:40:49.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:49.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:49.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:50.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:50.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:50.953 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:40:50.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:40:50.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:50.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:51.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:52.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:52.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:52.637 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:40:52.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:40:52.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:52.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:53.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:54.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:54.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:54.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:40:54.649 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:40:54.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:54.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:55.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:56.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:56.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:56.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:40:56.295 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:40:56.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:56.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:56.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:40:57.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:40:57.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:40:57.892 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:40:57.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:40:57.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:40:57.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:40:58.388 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3927573709,v1:127.0.0.1:6827/3927573709] [v2:127.0.0.1:6828/3927573709,v1:127.0.0.1:6829/3927573709] exists,up 7f7a3b9a-b289-4432-84a7-7b991ae08cbf 2024-09-17T16:40:58.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:40:58.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:40:58.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:40:58.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:40:58.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-17T16:40:58.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:40:58.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:40:58.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-17T16:40:58.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:40:58.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-17T16:40:58.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-17T16:40:58.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:40:58.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:40:58.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:40:58.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-17T16:40:58.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-17T16:40:58.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:40:58.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:40:58.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:40:58.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:40:58.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:40:58.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:40:58.395 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:40:58.395 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:40:58.395 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:40:58.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:40:58.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:40:58.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:40:58.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:40:58.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:40:58.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:40:58.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:40:58.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:40:58.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:40:58.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:40:58.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:40:58.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:40:58.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:40:58.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-17T16:40:58.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:40:58.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8ebe18c1-f19d-4b4b-b686-c2a08fd6509e 2024-09-17T16:40:58.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 8ebe18c1-f19d-4b4b-b686-c2a08fd6509e' 2024-09-17T16:40:58.396 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 8ebe18c1-f19d-4b4b-b686-c2a08fd6509e 2024-09-17T16:40:58.397 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:40:58.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAaselmgPu1GBAAk2fIlt0Qd0pBUuDsiLf5cw== 2024-09-17T16:40:58.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAaselmgPu1GBAAk2fIlt0Qd0pBUuDsiLf5cw=="}' 2024-09-17T16:40:58.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8ebe18c1-f19d-4b4b-b686-c2a08fd6509e -i td/test-erasure-eio/4/new.json 2024-09-17T16:40:59.095 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:40:59.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-17T16:40:59.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAaselmgPu1GBAAk2fIlt0Qd0pBUuDsiLf5cw== --osd-uuid 8ebe18c1-f19d-4b4b-b686-c2a08fd6509e 2024-09-17T16:40:59.145 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:40:59.143+0000 7fba33a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:40:59.147 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:40:59.145+0000 7fba33a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:40:59.148 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:40:59.146+0000 7fba33a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:40:59.148 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:40:59.146+0000 7fba33a18640 -1 bdev(0x563e8309fc00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-17T16:40:59.148 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:40:59.146+0000 7fba33a18640 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-17T16:41:05.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-17T16:41:05.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:41:05.464 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-17T16:41:05.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-17T16:41:05.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:41:06.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-17T16:41:06.009 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-17T16:41:06.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:41:06.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:41:06.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:41:06.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:41:06.044 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:41:06.042+0000 7fd175158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:41:06.068 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:41:06.066+0000 7fd175158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:41:06.069 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:41:06.067+0000 7fd175158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:41:06.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-17T16:41:06.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:41:06.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-17T16:41:06.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:41:06.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:41:06.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:06.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:41:06.451 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:41:06.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:06.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:06.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:06.930 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:41:06.928+0000 7fd175158640 -1 Falling back to public interface 2024-09-17T16:41:07.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:07.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:07.886 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:41:07.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:41:07.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:07.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:08.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:09.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:09.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:09.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:41:09.310 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:41:09.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:09.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:09.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:10.325 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:41:10.323+0000 7fd175158640 -1 osd.4 0 log_to_monitors true 2024-09-17T16:41:10.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:10.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:10.721 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:41:10.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:41:10.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:10.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:11.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:12.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:12.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:12.213 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:41:12.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:41:12.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:12.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:12.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:13.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:13.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:13.991 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:41:13.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:41:13.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:13.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:14.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:15.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:15.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:15.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:41:15.425 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:41:15.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:15.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:15.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:16.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:16.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:16.857 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:41:16.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:41:16.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:16.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:18.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:18.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:18.306 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:41:18.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:41:18.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:18.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:18.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:19.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:19.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:19.735 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:41:19.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:41:19.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:19.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:20.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:21.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:21.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:21.206 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:41:21.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:41:21.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:21.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:21.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:22.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:22.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:22.649 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:41:22.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:41:22.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:22.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:23.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:24.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:24.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:24.167 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:41:24.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:41:24.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:24.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:24.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:25.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:25.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:25.919 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:41:25.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:41:25.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:25.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:26.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:27.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:27.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:27.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:41:27.711 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:41:27.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:27.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:28.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:29.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:29.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:29.237 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:41:29.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:41:29.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:29.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:29.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:30.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:30.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:30.636 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:41:30.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:41:30.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:30.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:31.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:32.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:32.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:32.090 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:41:32.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:41:32.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:32.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:32.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:33.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:33.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:33.574 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:41:33.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:41:33.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:33.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:34.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:41:35.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:41:35.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:41:35.556 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-09-17T16:41:35.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T16:41:35.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:41:35.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:41:36.037 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/155529760,v1:127.0.0.1:6835/155529760] [v2:127.0.0.1:6836/155529760,v1:127.0.0.1:6837/155529760] exists,up 8ebe18c1-f19d-4b4b-b686-c2a08fd6509e 2024-09-17T16:41:36.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:41:36.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:41:36.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:41:36.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-17T16:41:36.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:41:36.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:41:36.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:41:36.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:41:36.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:41:36.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T16:41:36.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-17T16:41:36.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.0.asok log flush 2024-09-17T16:41:36.119 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:41:36.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-17T16:41:36.131 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:38:42.401+0000 7f4995611640 0 load: jerasure load: lrc 2024-09-17T16:41:36.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: CEPH_ARGS='--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:41:36.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:641: TEST_ec_recovery_unfound: local poolname=pool-jerasure 2024-09-17T16:41:36.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:642: TEST_ec_recovery_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-09-17T16:41:36.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:41:36.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-17T16:41:36.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-17T16:41:36.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-17T16:41:36.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-17T16:41:36.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-17T16:41:36.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-17T16:41:37.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-17T16:41:37.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-17T16:41:38.286 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-17T16:41:38.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:41:39.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-17T16:41:39.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:41:39.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:41:39.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:41:39.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:41:39.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:41:39.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:41:39.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:41:39.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:41:39.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:41:39.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:41:39.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:41:39.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:41:39.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:41:39.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:41:39.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:41:39.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:41:39.788 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:41:39.788 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:41:39.789 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:41:39.789 INFO:tasks.workunit.client.0.smithi162.stderr:4' 2024-09-17T16:41:39.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:41:39.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:41:39.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:41:39.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803812 2024-09-17T16:41:39.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803812 2024-09-17T16:41:39.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812' 2024-09-17T16:41:39.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:41:39.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:41:40.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574881 2024-09-17T16:41:40.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574881 2024-09-17T16:41:40.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-55834574881' 2024-09-17T16:41:40.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:41:40.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:41:40.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345943 2024-09-17T16:41:40.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345943 2024-09-17T16:41:40.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-55834574881 2-85899345943' 2024-09-17T16:41:40.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:41:40.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:41:40.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018895 2024-09-17T16:41:40.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018895 2024-09-17T16:41:40.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-55834574881 2-85899345943 3-128849018895' 2024-09-17T16:41:40.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:41:40.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659141 2024-09-17T16:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659141 2024-09-17T16:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-55834574881 2-85899345943 3-128849018895 4-176093659141' 2024-09-17T16:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:41:40.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803812 2024-09-17T16:41:40.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:41:40.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:41:40.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803812 2024-09-17T16:41:40.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:41:40.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803812 2024-09-17T16:41:40.559 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803812 2024-09-17T16:41:40.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803812' 2024-09-17T16:41:40.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:41:40.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803810 -lt 25769803812 2024-09-17T16:41:40.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:41:41.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:41:41.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:41:42.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803812 -lt 25769803812 2024-09-17T16:41:42.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:41:42.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574881 2024-09-17T16:41:42.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:41:42.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:41:42.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574881 2024-09-17T16:41:42.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:41:42.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574881 2024-09-17T16:41:42.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574881' 2024-09-17T16:41:42.411 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574881 2024-09-17T16:41:42.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:41:42.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574881 -lt 55834574881 2024-09-17T16:41:42.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:41:42.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345943 2024-09-17T16:41:42.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:41:42.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:41:42.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345943 2024-09-17T16:41:42.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:41:42.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345943 2024-09-17T16:41:42.887 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 85899345943 2024-09-17T16:41:42.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345943' 2024-09-17T16:41:42.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:41:43.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345944 -lt 85899345943 2024-09-17T16:41:43.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:41:43.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018895 2024-09-17T16:41:43.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:41:43.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:41:43.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018895 2024-09-17T16:41:43.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:41:43.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018895 2024-09-17T16:41:43.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018895' 2024-09-17T16:41:43.353 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 128849018895 2024-09-17T16:41:43.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:41:43.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018895 -lt 128849018895 2024-09-17T16:41:43.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:41:43.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-176093659141 2024-09-17T16:41:43.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:41:43.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:41:43.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-176093659141 2024-09-17T16:41:43.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:41:43.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659141 2024-09-17T16:41:43.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 176093659141' 2024-09-17T16:41:43.738 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 176093659141 2024-09-17T16:41:43.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:41:44.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659142 -lt 176093659141 2024-09-17T16:41:44.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:41:44.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:41:44.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:41:44.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:41:44.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:41:44.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:41:44.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:41:44.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:41:44.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:41:44.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:41:44.663 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:41:45.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:41:45.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:41:45.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:41:45.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:41:45.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:41:45.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:41:45.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:41:45.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:644: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-17T16:41:45.993 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T16:41:45.993 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:41:40.086336+0000 0'0 48:16 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-17T16:41:38.013866+0000 0'0 2024-09-17T16:41:38.013866+0000 0 0 periodic scrub scheduled @ 2024-09-19T01:57:15.284437+0000 0 0 2024-09-17T16:41:45.993 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:41:37.155844+0000 0'0 47:19 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-17T16:38:33.223892+0000 0'0 2024-09-17T16:38:33.223892+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:54:39.431021+0000 0 0 2024-09-17T16:41:45.993 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:41:37.072939+0000 0'0 48:114 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-17T16:38:33.223892+0000 0'0 2024-09-17T16:38:33.223892+0000 0 0 periodic scrub scheduled @ 2024-09-18T19:59:15.383987+0000 0 0 2024-09-17T16:41:45.994 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:41:37.351328+0000 0'0 47:19 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-17T16:38:33.223892+0000 0'0 2024-09-17T16:38:33.223892+0000 0 0 periodic scrub scheduled @ 2024-09-18T19:59:32.775384+0000 0 0 2024-09-17T16:41:45.994 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:40:21.820417+0000 0'0 48:76 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T16:38:33.223892+0000 0'0 2024-09-17T16:38:33.223892+0000 0 0 periodic scrub scheduled @ 2024-09-19T01:10:26.236939+0000 0 0 2024-09-17T16:41:45.994 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-09-17T16:41:45.994 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T16:41:45.994 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-09-17T16:41:46.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:646: TEST_ec_recovery_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-17T16:41:46.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:41:46.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:41:46.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-17T16:41:46.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:41:46.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:41:46.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:41:46.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:41:46.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:41:46.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:41:46.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:41:46.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:41:46.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:46.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: get_osds pool-jerasure myobject 2024-09-17T16:41:46.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:41:46.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-17T16:41:46.165 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-17T16:41:46.166 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:41:46.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:41:46.653 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:41:46.653 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:41:46.653 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:41:46.653 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:41:46.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2 2024-09-17T16:41:46.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-09-17T16:41:46.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: local -a initial_osds 2024-09-17T16:41:46.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:649: TEST_ec_recovery_unfound: local last_osd=2 2024-09-17T16:41:46.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-17T16:41:46.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:41:46.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:41:46.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:41:46.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:41:46.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:41:46.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:41:46.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: ceph osd down 2 2024-09-17T16:41:47.453 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already down. 2024-09-17T16:41:47.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: ceph osd out 2 2024-09-17T16:41:48.266 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already out. 2024-09-17T16:41:48.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:654: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-17T16:41:48.761 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T16:41:48.761 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 1 0 1 0 0 4096 0 0 1 0 1 active+undersized+degraded 2024-09-17T16:41:47.904668+0000 48'1 50:27 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-09-17T16:41:38.013866+0000 0'0 2024-09-17T16:41:38.013866+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:17:29.546814+0000 0 0 2024-09-17T16:41:48.761 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:41:37.155844+0000 0'0 48:21 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-17T16:38:33.223892+0000 0'0 2024-09-17T16:38:33.223892+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:54:39.431021+0000 0 0 2024-09-17T16:41:48.761 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:41:37.072939+0000 0'0 48:114 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-17T16:38:33.223892+0000 0'0 2024-09-17T16:38:33.223892+0000 0 0 periodic scrub scheduled @ 2024-09-18T19:59:15.383987+0000 0 0 2024-09-17T16:41:48.761 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:41:37.351328+0000 0'0 48:21 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-17T16:38:33.223892+0000 0'0 2024-09-17T16:38:33.223892+0000 0 0 periodic scrub scheduled @ 2024-09-18T19:59:32.775384+0000 0 0 2024-09-17T16:41:48.761 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:40:21.820417+0000 0'0 48:76 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T16:38:33.223892+0000 0'0 2024-09-17T16:38:33.223892+0000 0 0 periodic scrub scheduled @ 2024-09-19T01:10:26.236939+0000 0 0 2024-09-17T16:41:48.761 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-09-17T16:41:48.761 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T16:41:48.761 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-09-17T16:41:48.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:656: TEST_ec_recovery_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-09-17T16:41:48.773 INFO:tasks.workunit.client.0.smithi162.stderr:4+0 records in 2024-09-17T16:41:48.774 INFO:tasks.workunit.client.0.smithi162.stderr:4+0 records out 2024-09-17T16:41:48.774 INFO:tasks.workunit.client.0.smithi162.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 8.3856e-05 s, 48.8 MB/s 2024-09-17T16:41:48.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: seq 1 100 2024-09-17T16:41:48.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:48.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:48.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:48.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:49.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:49.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:49.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:49.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:50.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:50.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:50.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:50.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:50.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:50.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:50.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:50.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:51.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:51.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:51.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:51.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:51.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:51.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:51.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:51.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:52.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:52.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:52.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:52.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:52.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:52.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:52.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:52.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:53.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:53.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:53.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:53.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:53.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:53.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:53.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:53.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:54.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:54.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:54.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:54.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:54.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:54.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:54.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:54.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:55.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:55.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:55.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:55.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:55.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:55.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:55.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:55.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:56.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:56.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:56.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:56.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:56.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:56.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:56.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:56.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:57.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:57.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:57.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:57.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:57.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:57.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:57.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:57.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:58.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:58.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:58.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:58.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:58.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:58.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:58.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:58.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:59.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:59.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:59.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:59.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:59.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:59.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-09-17T16:41:59.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:41:59.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:00.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:00.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:00.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:00.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:00.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:00.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:01.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:01.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:01.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:01.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:01.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:01.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:01.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:01.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:02.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:02.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:02.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:02.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:02.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:02.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:03.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:03.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:03.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:03.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:03.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:03.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:03.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:03.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:03.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:03.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:04.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:04.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:04.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:04.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:04.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:04.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:04.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:04.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:05.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:05.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:05.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:05.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:05.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:05.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:05.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:05.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:06.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:06.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:06.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:06.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:06.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:06.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:06.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:06.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:07.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:07.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:07.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:07.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:07.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:07.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:07.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:07.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:08.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:08.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:08.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:08.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:08.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:08.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:09.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:09.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:09.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:09.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:09.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:09.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:09.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:09.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:10.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:10.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:10.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:10.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:10.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:10.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:10.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:10.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:11.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:11.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:11.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:11.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:11.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:11.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:11.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:11.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:12.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:12.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:12.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:12.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:12.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:12.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:12.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:12.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:13.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:13.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:13.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:13.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:13.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:13.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:14.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:14.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-09-17T16:42:14.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:662: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 0 2024-09-17T16:42:14.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T16:42:14.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T16:42:14.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T16:42:14.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T16:42:14.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T16:42:14.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T16:42:14.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj75 2024-09-17T16:42:14.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T16:42:14.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T16:42:14.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T16:42:14.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-17T16:42:14.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T16:42:14.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj75 2024-09-17T16:42:14.285 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:42:14.285 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj75 2024-09-17T16:42:14.285 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj75 2024-09-17T16:42:14.285 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:42:14.685 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:42:14.685 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:42:14.685 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:42:14.686 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:42:14.686 INFO:tasks.workunit.client.0.smithi162.stderr:2147483647' 2024-09-17T16:42:14.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2024-09-17T16:42:14.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-17T16:42:14.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T16:42:14.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-17T16:42:14.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T16:42:14.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-17T16:42:14.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T16:42:14.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-17T16:42:14.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T16:42:14.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-17T16:42:14.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T16:42:14.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T16:42:14.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T16:42:14.688 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-17T16:42:14.688 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-17T16:42:14.688 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-17T16:42:14.688 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:42:14.688 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:42:14.689 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:42:14.689 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.3.asok 2024-09-17T16:42:14.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-17T16:42:14.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T16:42:14.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T16:42:14.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T16:42:14.775 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-17T16:42:14.775 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-17T16:42:14.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-17T16:42:14.776 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:42:14.776 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:42:14.776 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:42:14.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.3.asok 2024-09-17T16:42:14.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T16:42:14.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.3.asok injectdataerr pool-jerasure obj75 0 2024-09-17T16:42:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:663: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 1 2024-09-17T16:42:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T16:42:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T16:42:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T16:42:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T16:42:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T16:42:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T16:42:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj75 2024-09-17T16:42:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T16:42:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T16:42:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T16:42:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-17T16:42:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T16:42:14.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj75 2024-09-17T16:42:14.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:42:14.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj75 2024-09-17T16:42:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj75 2024-09-17T16:42:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:42:15.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:42:15.261 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:42:15.262 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:42:15.262 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:42:15.262 INFO:tasks.workunit.client.0.smithi162.stderr:2147483647' 2024-09-17T16:42:15.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2024-09-17T16:42:15.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-17T16:42:15.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T16:42:15.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-17T16:42:15.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T16:42:15.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/1/type 2024-09-17T16:42:15.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T16:42:15.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-17T16:42:15.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T16:42:15.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-17T16:42:15.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T16:42:15.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T16:42:15.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T16:42:15.264 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-17T16:42:15.264 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T16:42:15.265 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T16:42:15.265 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:42:15.265 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:42:15.265 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:42:15.265 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.1.asok 2024-09-17T16:42:15.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-17T16:42:15.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T16:42:15.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T16:42:15.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T16:42:15.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-17T16:42:15.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T16:42:15.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T16:42:15.352 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:42:15.352 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:42:15.352 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:42:15.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.1.asok 2024-09-17T16:42:15.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T16:42:15.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.1.asok injectdataerr pool-jerasure obj75 1 2024-09-17T16:42:15.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:665: TEST_ec_recovery_unfound: activate_osd td/test-erasure-eio 2 2024-09-17T16:42:15.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:42:15.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:42:15.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T16:42:15.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:42:15.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:42:15.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:42:15.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:42:15.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:42:15.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:42:15.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:42:15.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:42:15.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:42:15.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:42:15.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:42:15.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:42:15.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:42:15.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:42:15.442 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:42:15.442 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:42:15.442 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:42:15.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:42:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:42:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:42:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:42:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:42:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:42:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:42:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:42:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:42:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:42:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:42:15.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:42:15.444 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T16:42:15.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T16:42:15.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:42:15.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-17T16:42:15.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T16:42:15.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:42:15.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:42:15.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:42:15.468 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:42:15.466+0000 7f644f1a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:42:15.469 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:42:15.468+0000 7f644f1a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:42:15.470 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:42:15.469+0000 7f644f1a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:42:15.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T16:42:15.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:42:15.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:42:15.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:42:15.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:42:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:42:15.866 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:42:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:42:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:42:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:42:16.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:42:16.564 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:42:16.562+0000 7f644f1a8640 -1 Falling back to public interface 2024-09-17T16:42:17.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:42:17.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:42:17.274 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:42:17.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:42:17.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:42:17.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:42:17.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:42:18.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:42:18.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:42:18.718 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:42:18.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:42:18.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:42:18.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:42:19.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:42:19.802 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:42:19.800+0000 7f644f1a8640 -1 osd.2 48 log_to_monitors true 2024-09-17T16:42:20.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:42:20.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:42:20.108 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:42:20.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:42:20.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:42:20.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:42:20.579 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up out weight 0 up_from 55 up_thru 22 down_at 49 last_clean_interval [20,48) [v2:127.0.0.1:6818/2677041477,v1:127.0.0.1:6819/2677041477] [v2:127.0.0.1:6820/2677041477,v1:127.0.0.1:6821/2677041477] exists,up f634ce53-bc95-449b-89fc-41431f8afc45 2024-09-17T16:42:20.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:42:20.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:42:20.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:42:20.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: ceph osd in 2 2024-09-17T16:42:21.141 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already in. 2024-09-17T16:42:21.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:668: TEST_ec_recovery_unfound: sleep 15 2024-09-17T16:42:36.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: seq 1 100 2024-09-17T16:42:36.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: for tmp in $(seq 1 100) 2024-09-17T16:42:36.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:671: TEST_ec_recovery_unfound: get_state 2.0 2024-09-17T16:42:36.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-17T16:42:36.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-17T16:42:36.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-17T16:42:36.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-17T16:42:36.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:671: TEST_ec_recovery_unfound: state=active+recovery_unfound+degraded 2024-09-17T16:42:36.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:672: TEST_ec_recovery_unfound: echo active+recovery_unfound+degraded 2024-09-17T16:42:36.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:672: TEST_ec_recovery_unfound: grep recovery_unfound 2024-09-17T16:42:36.545 INFO:tasks.workunit.client.0.smithi162.stdout:active+recovery_unfound+degraded 2024-09-17T16:42:36.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:673: TEST_ec_recovery_unfound: '[' 0 = 0 ']' 2024-09-17T16:42:36.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:674: TEST_ec_recovery_unfound: break 2024-09-17T16:42:36.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-17T16:42:36.909 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T16:42:36.909 INFO:tasks.workunit.client.0.smithi162.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-09-17T16:42:25.271319+0000 53'101 58:347 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-17T16:41:38.013866+0000 0'0 2024-09-17T16:41:38.013866+0000 0 0 no scrub is scheduled 0 0 2024-09-17T16:42:36.909 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:42:22.143126+0000 0'0 58:57 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-17T16:38:33.223892+0000 0'0 2024-09-17T16:38:33.223892+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:31:50.982765+0000 0 0 2024-09-17T16:42:36.910 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:41:37.072939+0000 0'0 58:134 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-17T16:38:33.223892+0000 0'0 2024-09-17T16:38:33.223892+0000 0 0 periodic scrub scheduled @ 2024-09-18T19:59:15.383987+0000 0 0 2024-09-17T16:42:36.910 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:41:37.351328+0000 0'0 58:40 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-17T16:38:33.223892+0000 0'0 2024-09-17T16:38:33.223892+0000 0 0 periodic scrub scheduled @ 2024-09-18T19:59:32.775384+0000 0 0 2024-09-17T16:42:36.910 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T16:42:22.252698+0000 0'0 58:112 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T16:38:33.223892+0000 0'0 2024-09-17T16:38:33.223892+0000 0 0 periodic scrub scheduled @ 2024-09-18T17:46:17.121867+0000 0 0 2024-09-17T16:42:36.910 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-09-17T16:42:36.910 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T16:42:36.910 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-09-17T16:42:36.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:681: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-17T16:42:37.029 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-09-17T16:42:37.029 INFO:tasks.workunit.client.0.smithi162.stdout: "num_missing": 1, 2024-09-17T16:42:37.030 INFO:tasks.workunit.client.0.smithi162.stdout: "num_unfound": 1, 2024-09-17T16:42:37.030 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": [ 2024-09-17T16:42:37.030 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.030 INFO:tasks.workunit.client.0.smithi162.stdout: "oid": { 2024-09-17T16:42:37.030 INFO:tasks.workunit.client.0.smithi162.stdout: "oid": "obj75", 2024-09-17T16:42:37.030 INFO:tasks.workunit.client.0.smithi162.stdout: "key": "", 2024-09-17T16:42:37.030 INFO:tasks.workunit.client.0.smithi162.stdout: "snapid": -2, 2024-09-17T16:42:37.030 INFO:tasks.workunit.client.0.smithi162.stdout: "hash": 3565005477, 2024-09-17T16:42:37.030 INFO:tasks.workunit.client.0.smithi162.stdout: "max": 0, 2024-09-17T16:42:37.030 INFO:tasks.workunit.client.0.smithi162.stdout: "pool": 2, 2024-09-17T16:42:37.030 INFO:tasks.workunit.client.0.smithi162.stdout: "namespace": "" 2024-09-17T16:42:37.030 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.030 INFO:tasks.workunit.client.0.smithi162.stdout: "need": "53'76", 2024-09-17T16:42:37.031 INFO:tasks.workunit.client.0.smithi162.stdout: "have": "0'0", 2024-09-17T16:42:37.031 INFO:tasks.workunit.client.0.smithi162.stdout: "flags": "none", 2024-09-17T16:42:37.031 INFO:tasks.workunit.client.0.smithi162.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-17T16:42:37.031 INFO:tasks.workunit.client.0.smithi162.stdout: "locations": [ 2024-09-17T16:42:37.031 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-17T16:42:37.031 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-17T16:42:37.031 INFO:tasks.workunit.client.0.smithi162.stdout: ] 2024-09-17T16:42:37.031 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:42:37.031 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.031 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "NotRecovering", 2024-09-17T16:42:37.031 INFO:tasks.workunit.client.0.smithi162.stdout: "available_might_have_unfound": true, 2024-09-17T16:42:37.031 INFO:tasks.workunit.client.0.smithi162.stdout: "might_have_unfound": [], 2024-09-17T16:42:37.031 INFO:tasks.workunit.client.0.smithi162.stdout: "more": false 2024-09-17T16:42:37.031 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-09-17T16:42:37.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: ceph pg 2.0 query 2024-09-17T16:42:37.148 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-09-17T16:42:37.148 INFO:tasks.workunit.client.0.smithi162.stdout: "snap_trimq": "[]", 2024-09-17T16:42:37.148 INFO:tasks.workunit.client.0.smithi162.stdout: "snap_trimq_len": 0, 2024-09-17T16:42:37.148 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+recovery_unfound+degraded", 2024-09-17T16:42:37.148 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch": 58, 2024-09-17T16:42:37.148 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-17T16:42:37.149 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:42:37.149 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:42:37.149 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:42:37.149 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:42:37.149 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-17T16:42:37.149 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.149 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-17T16:42:37.149 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:42:37.149 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:42:37.149 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:42:37.149 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:42:37.149 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-17T16:42:37.149 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.149 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_recovery_backfill": [ 2024-09-17T16:42:37.150 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-17T16:42:37.150 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-17T16:42:37.150 INFO:tasks.workunit.client.0.smithi162.stdout: "2(4)", 2024-09-17T16:42:37.150 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-17T16:42:37.150 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-17T16:42:37.150 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.150 INFO:tasks.workunit.client.0.smithi162.stdout: "info": { 2024-09-17T16:42:37.150 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s0", 2024-09-17T16:42:37.150 INFO:tasks.workunit.client.0.smithi162.stdout: "shared": "0", 2024-09-17T16:42:37.150 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "53'101", 2024-09-17T16:42:37.150 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "53'75", 2024-09-17T16:42:37.150 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "0'0", 2024-09-17T16:42:37.150 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 101, 2024-09-17T16:42:37.151 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-17T16:42:37.151 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-17T16:42:37.151 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-17T16:42:37.151 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 46, 2024-09-17T16:42:37.151 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 46, 2024-09-17T16:42:37.151 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 58, 2024-09-17T16:42:37.151 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 56, 2024-09-17T16:42:37.151 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:42:37.151 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 46, 2024-09-17T16:42:37.151 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-17T16:42:37.151 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-17T16:42:37.151 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 56, 2024-09-17T16:42:37.151 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 56, 2024-09-17T16:42:37.151 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 46, 2024-09-17T16:42:37.152 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:42:37.152 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.152 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:42:37.152 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.152 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.152 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-17T16:42:37.152 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.152 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-17T16:42:37.152 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "53'101", 2024-09-17T16:42:37.152 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 348, 2024-09-17T16:42:37.152 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 58, 2024-09-17T16:42:37.152 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+recovery_unfound+degraded", 2024-09-17T16:42:37.152 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-17T16:42:25.362546+0000", 2024-09-17T16:42:37.152 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-17T16:42:25.271319+0000", 2024-09-17T16:42:37.153 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-17T16:42:25.362546+0000", 2024-09-17T16:42:37.153 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-17T16:42:25.362546+0000", 2024-09-17T16:42:37.153 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-17T16:41:46.160865+0000", 2024-09-17T16:42:37.153 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-17T16:42:22.353237+0000", 2024-09-17T16:42:37.153 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-17T16:42:22.353237+0000", 2024-09-17T16:42:37.153 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-17T16:42:25.362546+0000", 2024-09-17T16:42:37.153 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-17T16:42:21.913623+0000", 2024-09-17T16:42:37.153 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-17T16:42:25.362546+0000", 2024-09-17T16:42:37.153 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 56, 2024-09-17T16:42:37.153 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "0'0", 2024-09-17T16:42:37.153 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "0'0", 2024-09-17T16:42:37.153 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 46, 2024-09-17T16:42:37.153 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:42:37.153 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-17T16:42:37.154 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-17T16:42:37.154 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:42:37.154 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.154 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:42:37.154 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.154 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.154 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-17T16:42:37.154 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-17T16:42:37.154 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 0, 2024-09-17T16:42:37.154 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-17T16:42:37.154 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-17T16:42:37.154 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-17T16:42:37.154 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-17T16:42:37.154 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-17T16:42:37.155 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-17T16:42:37.155 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-17T16:42:37.155 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-17T16:42:37.155 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-17T16:42:37.155 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-17T16:42:37.155 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "no scrub is scheduled", 2024-09-17T16:42:37.155 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-17T16:42:37.155 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-17T16:42:37.155 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-17T16:42:37.155 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-17T16:42:37.155 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 413696, 2024-09-17T16:42:37.155 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 101, 2024-09-17T16:42:37.155 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-17T16:42:37.155 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 505, 2024-09-17T16:42:37.156 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 1, 2024-09-17T16:42:37.156 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 0, 2024-09-17T16:42:37.156 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 3, 2024-09-17T16:42:37.156 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-17T16:42:37.156 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 1, 2024-09-17T16:42:37.156 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 101, 2024-09-17T16:42:37.156 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-17T16:42:37.156 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-17T16:42:37.156 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-17T16:42:37.156 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 101, 2024-09-17T16:42:37.157 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 404, 2024-09-17T16:42:37.157 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-17T16:42:37.157 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T16:42:37.157 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-17T16:42:37.157 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 99, 2024-09-17T16:42:37.157 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 405504, 2024-09-17T16:42:37.157 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-17T16:42:37.157 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-17T16:42:37.157 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T16:42:37.157 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T16:42:37.157 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-17T16:42:37.157 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-17T16:42:37.157 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-17T16:42:37.157 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-17T16:42:37.158 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-17T16:42:37.158 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-17T16:42:37.158 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-17T16:42:37.158 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-17T16:42:37.158 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-17T16:42:37.158 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-17T16:42:37.158 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-17T16:42:37.158 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-17T16:42:37.158 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-17T16:42:37.158 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-17T16:42:37.158 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-17T16:42:37.158 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-17T16:42:37.158 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.158 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-17T16:42:37.159 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:42:37.159 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:42:37.159 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:42:37.159 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:42:37.159 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-17T16:42:37.159 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.159 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-17T16:42:37.159 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:42:37.159 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:42:37.159 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:42:37.159 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:42:37.159 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-17T16:42:37.159 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.159 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-17T16:42:37.160 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-17T16:42:37.160 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-17T16:42:37.160 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.160 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-17T16:42:37.160 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.160 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),2(4),3(0),4(2)", 2024-09-17T16:42:37.160 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 100 2024-09-17T16:42:37.160 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.160 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.160 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),4(2)", 2024-09-17T16:42:37.160 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 1 2024-09-17T16:42:37.160 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:42:37.160 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.160 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-17T16:42:37.161 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-17T16:42:37.161 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-17T16:42:37.161 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-17T16:42:37.161 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.161 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-17T16:42:37.161 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-17T16:42:37.161 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-17T16:42:37.161 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 58, 2024-09-17T16:42:37.161 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-17T16:42:37.161 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-17T16:42:37.161 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-17T16:42:37.161 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:42:37.161 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.161 INFO:tasks.workunit.client.0.smithi162.stdout: "peer_info": [ 2024-09-17T16:42:37.162 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.162 INFO:tasks.workunit.client.0.smithi162.stdout: "peer": "0(3)", 2024-09-17T16:42:37.162 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s3", 2024-09-17T16:42:37.162 INFO:tasks.workunit.client.0.smithi162.stdout: "shared": "3", 2024-09-17T16:42:37.162 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "53'101", 2024-09-17T16:42:37.162 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "53'101", 2024-09-17T16:42:37.162 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "0'0", 2024-09-17T16:42:37.162 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 101, 2024-09-17T16:42:37.162 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-17T16:42:37.162 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-17T16:42:37.162 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-17T16:42:37.162 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 46, 2024-09-17T16:42:37.162 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 46, 2024-09-17T16:42:37.162 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 58, 2024-09-17T16:42:37.163 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 56, 2024-09-17T16:42:37.163 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:42:37.163 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 46, 2024-09-17T16:42:37.163 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-17T16:42:37.163 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-17T16:42:37.163 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 56, 2024-09-17T16:42:37.163 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 56, 2024-09-17T16:42:37.163 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 46, 2024-09-17T16:42:37.163 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:42:37.163 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.163 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:42:37.163 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.163 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.163 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-17T16:42:37.163 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.164 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-17T16:42:37.164 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "53'101", 2024-09-17T16:42:37.164 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 134, 2024-09-17T16:42:37.164 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 53, 2024-09-17T16:42:37.164 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+undersized+degraded", 2024-09-17T16:42:37.164 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-17T16:42:14.278419+0000", 2024-09-17T16:42:37.164 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-17T16:41:47.904668+0000", 2024-09-17T16:42:37.164 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-17T16:42:14.278419+0000", 2024-09-17T16:42:37.164 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-17T16:42:14.278419+0000", 2024-09-17T16:42:37.164 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-17T16:41:46.160865+0000", 2024-09-17T16:42:37.164 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-17T16:41:47.904668+0000", 2024-09-17T16:42:37.164 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-17T16:41:47.904668+0000", 2024-09-17T16:42:37.164 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-17T16:42:14.278419+0000", 2024-09-17T16:42:37.165 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-17T16:41:47.494026+0000", 2024-09-17T16:42:37.165 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-17T16:41:47.493826+0000", 2024-09-17T16:42:37.165 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 56, 2024-09-17T16:42:37.165 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "0'0", 2024-09-17T16:42:37.165 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "0'0", 2024-09-17T16:42:37.165 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 46, 2024-09-17T16:42:37.165 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:42:37.165 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-17T16:42:37.165 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-17T16:42:37.165 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:42:37.165 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.165 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:42:37.165 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.165 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.166 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-17T16:42:37.166 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-17T16:42:37.166 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 0, 2024-09-17T16:42:37.166 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-17T16:42:37.166 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-17T16:42:37.166 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-17T16:42:37.166 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-17T16:42:37.166 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-17T16:42:37.166 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-17T16:42:37.166 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-17T16:42:37.166 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-17T16:42:37.166 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-17T16:42:37.166 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-17T16:42:37.166 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-19T00:17:29.546814+0000", 2024-09-17T16:42:37.167 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-17T16:42:37.167 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-17T16:42:37.167 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-17T16:42:37.167 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-17T16:42:37.167 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 413696, 2024-09-17T16:42:37.167 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 101, 2024-09-17T16:42:37.167 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-17T16:42:37.167 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 505, 2024-09-17T16:42:37.167 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T16:42:37.167 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 0, 2024-09-17T16:42:37.167 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 101, 2024-09-17T16:42:37.167 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-17T16:42:37.167 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-09-17T16:42:37.167 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 101, 2024-09-17T16:42:37.167 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-17T16:42:37.168 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-17T16:42:37.168 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-17T16:42:37.168 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 101, 2024-09-17T16:42:37.168 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 404, 2024-09-17T16:42:37.168 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-17T16:42:37.168 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T16:42:37.168 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-17T16:42:37.168 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-17T16:42:37.168 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-17T16:42:37.168 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-17T16:42:37.168 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-17T16:42:37.169 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T16:42:37.169 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T16:42:37.169 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-17T16:42:37.169 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-17T16:42:37.169 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-17T16:42:37.169 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-17T16:42:37.169 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-17T16:42:37.169 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-17T16:42:37.169 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-17T16:42:37.169 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-17T16:42:37.169 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-17T16:42:37.169 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-17T16:42:37.169 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-17T16:42:37.169 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-17T16:42:37.169 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-17T16:42:37.170 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-17T16:42:37.170 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-17T16:42:37.170 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-17T16:42:37.170 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.170 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-17T16:42:37.170 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:42:37.170 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:42:37.170 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:42:37.170 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:42:37.170 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-17T16:42:37.170 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.170 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-17T16:42:37.170 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:42:37.170 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:42:37.171 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:42:37.171 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:42:37.171 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-17T16:42:37.171 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.171 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-17T16:42:37.171 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-17T16:42:37.171 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-17T16:42:37.171 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-17T16:42:37.171 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-17T16:42:37.171 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.171 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-17T16:42:37.171 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.171 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-17T16:42:37.171 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 101 2024-09-17T16:42:37.171 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:42:37.172 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.172 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-17T16:42:37.172 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-17T16:42:37.172 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-17T16:42:37.172 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-17T16:42:37.172 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.172 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-17T16:42:37.172 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-17T16:42:37.172 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-17T16:42:37.172 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 58, 2024-09-17T16:42:37.172 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-17T16:42:37.172 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-17T16:42:37.173 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-17T16:42:37.173 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:42:37.173 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.173 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.173 INFO:tasks.workunit.client.0.smithi162.stdout: "peer": "1(1)", 2024-09-17T16:42:37.173 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s1", 2024-09-17T16:42:37.173 INFO:tasks.workunit.client.0.smithi162.stdout: "shared": "1", 2024-09-17T16:42:37.173 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "53'101", 2024-09-17T16:42:37.173 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "53'101", 2024-09-17T16:42:37.173 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "0'0", 2024-09-17T16:42:37.173 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 101, 2024-09-17T16:42:37.173 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-17T16:42:37.173 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-17T16:42:37.173 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-17T16:42:37.174 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 46, 2024-09-17T16:42:37.174 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 46, 2024-09-17T16:42:37.174 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 58, 2024-09-17T16:42:37.174 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 56, 2024-09-17T16:42:37.174 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:42:37.174 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 46, 2024-09-17T16:42:37.174 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-17T16:42:37.174 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-17T16:42:37.174 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 56, 2024-09-17T16:42:37.174 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 56, 2024-09-17T16:42:37.174 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 46, 2024-09-17T16:42:37.174 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:42:37.174 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.174 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:42:37.174 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.175 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.175 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-17T16:42:37.175 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.175 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-17T16:42:37.175 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "53'101", 2024-09-17T16:42:37.175 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 134, 2024-09-17T16:42:37.175 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 53, 2024-09-17T16:42:37.175 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+undersized+degraded", 2024-09-17T16:42:37.175 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-17T16:42:14.278419+0000", 2024-09-17T16:42:37.175 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-17T16:41:47.904668+0000", 2024-09-17T16:42:37.175 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-17T16:42:14.278419+0000", 2024-09-17T16:42:37.175 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-17T16:42:14.278419+0000", 2024-09-17T16:42:37.175 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-17T16:41:46.160865+0000", 2024-09-17T16:42:37.175 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-17T16:41:47.904668+0000", 2024-09-17T16:42:37.175 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-17T16:41:47.904668+0000", 2024-09-17T16:42:37.176 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-17T16:42:14.278419+0000", 2024-09-17T16:42:37.176 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-17T16:41:47.494026+0000", 2024-09-17T16:42:37.176 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-17T16:41:47.493826+0000", 2024-09-17T16:42:37.176 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 56, 2024-09-17T16:42:37.176 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "0'0", 2024-09-17T16:42:37.176 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "0'0", 2024-09-17T16:42:37.176 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 46, 2024-09-17T16:42:37.176 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:42:37.176 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-17T16:42:37.176 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-17T16:42:37.176 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:42:37.176 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.176 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:42:37.177 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.177 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.177 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-17T16:42:37.177 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-17T16:42:37.177 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 0, 2024-09-17T16:42:37.177 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-17T16:42:37.178 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-17T16:42:37.178 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-17T16:42:37.178 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-17T16:42:37.178 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-17T16:42:37.178 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-17T16:42:37.178 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-17T16:42:37.178 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-17T16:42:37.179 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-17T16:42:37.179 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-17T16:42:37.179 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-19T00:17:29.546814+0000", 2024-09-17T16:42:37.179 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-17T16:42:37.179 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-17T16:42:37.179 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-17T16:42:37.179 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-17T16:42:37.179 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 413696, 2024-09-17T16:42:37.180 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 101, 2024-09-17T16:42:37.180 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-17T16:42:37.180 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 505, 2024-09-17T16:42:37.180 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T16:42:37.180 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 1, 2024-09-17T16:42:37.180 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 101, 2024-09-17T16:42:37.181 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-17T16:42:37.181 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-09-17T16:42:37.181 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 101, 2024-09-17T16:42:37.181 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-17T16:42:37.181 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-17T16:42:37.181 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-17T16:42:37.182 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 101, 2024-09-17T16:42:37.182 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 404, 2024-09-17T16:42:37.182 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-17T16:42:37.182 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T16:42:37.182 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-17T16:42:37.182 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-17T16:42:37.183 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-17T16:42:37.183 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-17T16:42:37.183 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-17T16:42:37.183 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T16:42:37.183 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T16:42:37.183 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-17T16:42:37.184 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-17T16:42:37.184 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-17T16:42:37.184 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-17T16:42:37.184 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-17T16:42:37.184 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-17T16:42:37.185 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-17T16:42:37.185 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-17T16:42:37.185 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-17T16:42:37.185 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-17T16:42:37.185 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-17T16:42:37.185 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-17T16:42:37.186 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-17T16:42:37.186 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-17T16:42:37.186 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-17T16:42:37.186 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-17T16:42:37.186 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.186 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-17T16:42:37.187 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:42:37.187 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:42:37.187 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:42:37.187 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:42:37.187 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-17T16:42:37.187 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.188 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-17T16:42:37.188 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:42:37.188 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:42:37.188 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:42:37.188 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:42:37.189 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-17T16:42:37.193 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.193 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-17T16:42:37.193 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-17T16:42:37.193 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-17T16:42:37.193 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-17T16:42:37.193 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-17T16:42:37.194 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.194 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-17T16:42:37.194 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.194 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-17T16:42:37.194 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 101 2024-09-17T16:42:37.195 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:42:37.195 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.195 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-17T16:42:37.195 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-17T16:42:37.195 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-17T16:42:37.195 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-17T16:42:37.196 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.196 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-17T16:42:37.196 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-17T16:42:37.196 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-17T16:42:37.196 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 58, 2024-09-17T16:42:37.197 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-17T16:42:37.197 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-17T16:42:37.197 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-17T16:42:37.197 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:42:37.197 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.197 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.198 INFO:tasks.workunit.client.0.smithi162.stdout: "peer": "2(4)", 2024-09-17T16:42:37.198 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s4", 2024-09-17T16:42:37.198 INFO:tasks.workunit.client.0.smithi162.stdout: "shared": "4", 2024-09-17T16:42:37.198 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "53'101", 2024-09-17T16:42:37.198 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "48'1", 2024-09-17T16:42:37.198 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "0'0", 2024-09-17T16:42:37.199 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 1, 2024-09-17T16:42:37.199 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-17T16:42:37.199 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-17T16:42:37.199 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-17T16:42:37.199 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 46, 2024-09-17T16:42:37.200 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 46, 2024-09-17T16:42:37.200 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 58, 2024-09-17T16:42:37.200 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 56, 2024-09-17T16:42:37.200 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:42:37.200 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 46, 2024-09-17T16:42:37.200 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-17T16:42:37.201 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-17T16:42:37.201 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 56, 2024-09-17T16:42:37.201 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 56, 2024-09-17T16:42:37.201 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 46, 2024-09-17T16:42:37.201 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:42:37.201 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.202 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:42:37.202 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.202 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.202 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-17T16:42:37.202 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.203 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-17T16:42:37.203 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "48'1", 2024-09-17T16:42:37.203 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 18, 2024-09-17T16:42:37.203 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 48, 2024-09-17T16:42:37.203 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+clean", 2024-09-17T16:42:37.203 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-17T16:41:46.160865+0000", 2024-09-17T16:42:37.204 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-17T16:41:40.086336+0000", 2024-09-17T16:42:37.204 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-17T16:41:46.160865+0000", 2024-09-17T16:42:37.204 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-17T16:41:46.160865+0000", 2024-09-17T16:42:37.204 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-17T16:41:46.160865+0000", 2024-09-17T16:42:37.204 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-17T16:41:40.085819+0000", 2024-09-17T16:42:37.204 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-17T16:41:40.085819+0000", 2024-09-17T16:42:37.204 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-17T16:41:46.160865+0000", 2024-09-17T16:42:37.205 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-17T16:41:46.160865+0000", 2024-09-17T16:42:37.205 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-17T16:41:46.160865+0000", 2024-09-17T16:42:37.205 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 56, 2024-09-17T16:42:37.205 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "0'0", 2024-09-17T16:42:37.205 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "0'0", 2024-09-17T16:42:37.205 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 46, 2024-09-17T16:42:37.206 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:42:37.206 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-17T16:42:37.206 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-17T16:42:37.206 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:42:37.206 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.206 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:42:37.207 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.207 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.207 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-17T16:42:37.207 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 1, 2024-09-17T16:42:37.207 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 0, 2024-09-17T16:42:37.207 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 1, 2024-09-17T16:42:37.208 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-17T16:42:37.208 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-17T16:42:37.208 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-17T16:42:37.208 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-17T16:42:37.208 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-17T16:42:37.209 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-17T16:42:37.209 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-17T16:42:37.209 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-17T16:42:37.209 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-17T16:42:37.209 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-19T01:57:15.284437+0000", 2024-09-17T16:42:37.209 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-17T16:42:37.210 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-17T16:42:37.210 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-17T16:42:37.210 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-17T16:42:37.210 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 4096, 2024-09-17T16:42:37.210 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 1, 2024-09-17T16:42:37.210 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-17T16:42:37.211 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 5, 2024-09-17T16:42:37.211 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T16:42:37.211 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 1, 2024-09-17T16:42:37.211 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 0, 2024-09-17T16:42:37.211 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-17T16:42:37.211 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-09-17T16:42:37.212 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 1, 2024-09-17T16:42:37.212 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-17T16:42:37.212 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-17T16:42:37.212 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-17T16:42:37.212 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 1, 2024-09-17T16:42:37.212 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 4, 2024-09-17T16:42:37.213 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-17T16:42:37.213 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T16:42:37.213 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-17T16:42:37.213 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-17T16:42:37.213 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-17T16:42:37.213 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-17T16:42:37.214 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-17T16:42:37.214 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T16:42:37.214 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T16:42:37.214 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-17T16:42:37.214 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-17T16:42:37.214 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-17T16:42:37.215 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-17T16:42:37.215 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-17T16:42:37.215 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-17T16:42:37.215 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-17T16:42:37.215 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-17T16:42:37.215 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-17T16:42:37.216 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-17T16:42:37.216 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-17T16:42:37.216 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-17T16:42:37.216 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-17T16:42:37.216 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-17T16:42:37.216 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-17T16:42:37.217 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-17T16:42:37.217 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.217 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-17T16:42:37.217 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:42:37.217 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:42:37.217 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:42:37.218 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:42:37.218 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-17T16:42:37.218 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.218 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-17T16:42:37.218 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:42:37.218 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:42:37.219 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:42:37.219 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:42:37.219 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-17T16:42:37.219 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.219 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [], 2024-09-17T16:42:37.219 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [], 2024-09-17T16:42:37.220 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-17T16:42:37.220 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-17T16:42:37.220 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-17T16:42:37.220 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-17T16:42:37.220 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.220 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-17T16:42:37.221 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-17T16:42:37.221 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-17T16:42:37.221 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 58, 2024-09-17T16:42:37.221 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-17T16:42:37.221 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-17T16:42:37.221 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-17T16:42:37.222 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:42:37.222 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.222 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.222 INFO:tasks.workunit.client.0.smithi162.stdout: "peer": "4(2)", 2024-09-17T16:42:37.222 INFO:tasks.workunit.client.0.smithi162.stdout: "pgid": "2.0s2", 2024-09-17T16:42:37.222 INFO:tasks.workunit.client.0.smithi162.stdout: "shared": "2", 2024-09-17T16:42:37.223 INFO:tasks.workunit.client.0.smithi162.stdout: "last_update": "53'101", 2024-09-17T16:42:37.223 INFO:tasks.workunit.client.0.smithi162.stdout: "last_complete": "53'101", 2024-09-17T16:42:37.223 INFO:tasks.workunit.client.0.smithi162.stdout: "log_tail": "0'0", 2024-09-17T16:42:37.223 INFO:tasks.workunit.client.0.smithi162.stdout: "last_user_version": 101, 2024-09-17T16:42:37.223 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill": "MAX", 2024-09-17T16:42:37.223 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [], 2024-09-17T16:42:37.224 INFO:tasks.workunit.client.0.smithi162.stdout: "history": { 2024-09-17T16:42:37.224 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_created": 46, 2024-09-17T16:42:37.224 INFO:tasks.workunit.client.0.smithi162.stdout: "epoch_pool_created": 46, 2024-09-17T16:42:37.224 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 58, 2024-09-17T16:42:37.224 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_started": 56, 2024-09-17T16:42:37.224 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:42:37.225 INFO:tasks.workunit.client.0.smithi162.stdout: "last_interval_clean": 46, 2024-09-17T16:42:37.225 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_split": 0, 2024-09-17T16:42:37.225 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_marked_full": 0, 2024-09-17T16:42:37.225 INFO:tasks.workunit.client.0.smithi162.stdout: "same_up_since": 56, 2024-09-17T16:42:37.225 INFO:tasks.workunit.client.0.smithi162.stdout: "same_interval_since": 56, 2024-09-17T16:42:37.225 INFO:tasks.workunit.client.0.smithi162.stdout: "same_primary_since": 46, 2024-09-17T16:42:37.226 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:42:37.226 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.226 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:42:37.226 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.226 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.226 INFO:tasks.workunit.client.0.smithi162.stdout: "prior_readable_until_ub": 0 2024-09-17T16:42:37.226 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.227 INFO:tasks.workunit.client.0.smithi162.stdout: "stats": { 2024-09-17T16:42:37.227 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "53'101", 2024-09-17T16:42:37.227 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_seq": 134, 2024-09-17T16:42:37.227 INFO:tasks.workunit.client.0.smithi162.stdout: "reported_epoch": 53, 2024-09-17T16:42:37.227 INFO:tasks.workunit.client.0.smithi162.stdout: "state": "active+undersized+degraded", 2024-09-17T16:42:37.227 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fresh": "2024-09-17T16:42:14.278419+0000", 2024-09-17T16:42:37.228 INFO:tasks.workunit.client.0.smithi162.stdout: "last_change": "2024-09-17T16:41:47.904668+0000", 2024-09-17T16:42:37.228 INFO:tasks.workunit.client.0.smithi162.stdout: "last_active": "2024-09-17T16:42:14.278419+0000", 2024-09-17T16:42:37.228 INFO:tasks.workunit.client.0.smithi162.stdout: "last_peered": "2024-09-17T16:42:14.278419+0000", 2024-09-17T16:42:37.228 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean": "2024-09-17T16:41:46.160865+0000", 2024-09-17T16:42:37.228 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_active": "2024-09-17T16:41:47.904668+0000", 2024-09-17T16:42:37.228 INFO:tasks.workunit.client.0.smithi162.stdout: "last_became_peered": "2024-09-17T16:41:47.904668+0000", 2024-09-17T16:42:37.229 INFO:tasks.workunit.client.0.smithi162.stdout: "last_unstale": "2024-09-17T16:42:14.278419+0000", 2024-09-17T16:42:37.229 INFO:tasks.workunit.client.0.smithi162.stdout: "last_undegraded": "2024-09-17T16:41:47.494026+0000", 2024-09-17T16:42:37.229 INFO:tasks.workunit.client.0.smithi162.stdout: "last_fullsized": "2024-09-17T16:41:47.493826+0000", 2024-09-17T16:42:37.229 INFO:tasks.workunit.client.0.smithi162.stdout: "mapping_epoch": 56, 2024-09-17T16:42:37.229 INFO:tasks.workunit.client.0.smithi162.stdout: "log_start": "0'0", 2024-09-17T16:42:37.229 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_start": "0'0", 2024-09-17T16:42:37.230 INFO:tasks.workunit.client.0.smithi162.stdout: "created": 46, 2024-09-17T16:42:37.230 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_clean": 47, 2024-09-17T16:42:37.230 INFO:tasks.workunit.client.0.smithi162.stdout: "parent": "0.0", 2024-09-17T16:42:37.230 INFO:tasks.workunit.client.0.smithi162.stdout: "parent_split_bits": 0, 2024-09-17T16:42:37.230 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub": "0'0", 2024-09-17T16:42:37.230 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.231 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub": "0'0", 2024-09-17T16:42:37.231 INFO:tasks.workunit.client.0.smithi162.stdout: "last_deep_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.231 INFO:tasks.workunit.client.0.smithi162.stdout: "last_clean_scrub_stamp": "2024-09-17T16:41:38.013866+0000", 2024-09-17T16:42:37.231 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_scrubbed": 0, 2024-09-17T16:42:37.231 INFO:tasks.workunit.client.0.smithi162.stdout: "log_size": 101, 2024-09-17T16:42:37.231 INFO:tasks.workunit.client.0.smithi162.stdout: "log_dups_size": 0, 2024-09-17T16:42:37.232 INFO:tasks.workunit.client.0.smithi162.stdout: "ondisk_log_size": 101, 2024-09-17T16:42:37.232 INFO:tasks.workunit.client.0.smithi162.stdout: "stats_invalid": false, 2024-09-17T16:42:37.232 INFO:tasks.workunit.client.0.smithi162.stdout: "dirty_stats_invalid": false, 2024-09-17T16:42:37.232 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_stats_invalid": false, 2024-09-17T16:42:37.232 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_stats_invalid": false, 2024-09-17T16:42:37.232 INFO:tasks.workunit.client.0.smithi162.stdout: "hitset_bytes_stats_invalid": false, 2024-09-17T16:42:37.232 INFO:tasks.workunit.client.0.smithi162.stdout: "pin_stats_invalid": false, 2024-09-17T16:42:37.233 INFO:tasks.workunit.client.0.smithi162.stdout: "manifest_stats_invalid": false, 2024-09-17T16:42:37.233 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrimq_len": 0, 2024-09-17T16:42:37.233 INFO:tasks.workunit.client.0.smithi162.stdout: "last_scrub_duration": 0, 2024-09-17T16:42:37.233 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-19T00:17:29.546814+0000", 2024-09-17T16:42:37.233 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_duration": 0, 2024-09-17T16:42:37.233 INFO:tasks.workunit.client.0.smithi162.stdout: "objects_trimmed": 0, 2024-09-17T16:42:37.234 INFO:tasks.workunit.client.0.smithi162.stdout: "snaptrim_duration": 0, 2024-09-17T16:42:37.234 INFO:tasks.workunit.client.0.smithi162.stdout: "stat_sum": { 2024-09-17T16:42:37.234 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 413696, 2024-09-17T16:42:37.234 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 101, 2024-09-17T16:42:37.234 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-09-17T16:42:37.234 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 505, 2024-09-17T16:42:37.235 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T16:42:37.235 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 0, 2024-09-17T16:42:37.235 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 101, 2024-09-17T16:42:37.235 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-09-17T16:42:37.235 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-09-17T16:42:37.236 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 101, 2024-09-17T16:42:37.236 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-09-17T16:42:37.236 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-09-17T16:42:37.236 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-09-17T16:42:37.236 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 101, 2024-09-17T16:42:37.236 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 404, 2024-09-17T16:42:37.236 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-09-17T16:42:37.236 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T16:42:37.237 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-09-17T16:42:37.237 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 0, 2024-09-17T16:42:37.237 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 0, 2024-09-17T16:42:37.237 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-09-17T16:42:37.237 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-09-17T16:42:37.237 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T16:42:37.237 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T16:42:37.237 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-09-17T16:42:37.237 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-09-17T16:42:37.238 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-09-17T16:42:37.238 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-09-17T16:42:37.238 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-09-17T16:42:37.238 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-09-17T16:42:37.238 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-09-17T16:42:37.238 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-09-17T16:42:37.238 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-09-17T16:42:37.239 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-09-17T16:42:37.239 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-09-17T16:42:37.239 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-09-17T16:42:37.239 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-09-17T16:42:37.239 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-09-17T16:42:37.239 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-09-17T16:42:37.239 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 0 2024-09-17T16:42:37.240 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.240 INFO:tasks.workunit.client.0.smithi162.stdout: "up": [ 2024-09-17T16:42:37.240 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:42:37.240 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:42:37.240 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:42:37.240 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:42:37.240 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-17T16:42:37.240 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.240 INFO:tasks.workunit.client.0.smithi162.stdout: "acting": [ 2024-09-17T16:42:37.240 INFO:tasks.workunit.client.0.smithi162.stdout: 3, 2024-09-17T16:42:37.241 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-09-17T16:42:37.241 INFO:tasks.workunit.client.0.smithi162.stdout: 4, 2024-09-17T16:42:37.241 INFO:tasks.workunit.client.0.smithi162.stdout: 0, 2024-09-17T16:42:37.241 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-09-17T16:42:37.241 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.241 INFO:tasks.workunit.client.0.smithi162.stdout: "avail_no_missing": [ 2024-09-17T16:42:37.241 INFO:tasks.workunit.client.0.smithi162.stdout: "3(0)", 2024-09-17T16:42:37.241 INFO:tasks.workunit.client.0.smithi162.stdout: "0(3)", 2024-09-17T16:42:37.241 INFO:tasks.workunit.client.0.smithi162.stdout: "1(1)", 2024-09-17T16:42:37.242 INFO:tasks.workunit.client.0.smithi162.stdout: "4(2)" 2024-09-17T16:42:37.242 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.242 INFO:tasks.workunit.client.0.smithi162.stdout: "object_location_counts": [ 2024-09-17T16:42:37.242 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.242 INFO:tasks.workunit.client.0.smithi162.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-17T16:42:37.242 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": 101 2024-09-17T16:42:37.242 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:42:37.242 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.242 INFO:tasks.workunit.client.0.smithi162.stdout: "blocked_by": [], 2024-09-17T16:42:37.242 INFO:tasks.workunit.client.0.smithi162.stdout: "up_primary": 3, 2024-09-17T16:42:37.242 INFO:tasks.workunit.client.0.smithi162.stdout: "acting_primary": 3, 2024-09-17T16:42:37.243 INFO:tasks.workunit.client.0.smithi162.stdout: "purged_snaps": [] 2024-09-17T16:42:37.243 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.243 INFO:tasks.workunit.client.0.smithi162.stdout: "empty": 0, 2024-09-17T16:42:37.243 INFO:tasks.workunit.client.0.smithi162.stdout: "dne": 0, 2024-09-17T16:42:37.243 INFO:tasks.workunit.client.0.smithi162.stdout: "incomplete": 0, 2024-09-17T16:42:37.243 INFO:tasks.workunit.client.0.smithi162.stdout: "last_epoch_started": 58, 2024-09-17T16:42:37.244 INFO:tasks.workunit.client.0.smithi162.stdout: "hit_set_history": { 2024-09-17T16:42:37.244 INFO:tasks.workunit.client.0.smithi162.stdout: "current_last_update": "0'0", 2024-09-17T16:42:37.244 INFO:tasks.workunit.client.0.smithi162.stdout: "history": [] 2024-09-17T16:42:37.244 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:42:37.244 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:42:37.244 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.245 INFO:tasks.workunit.client.0.smithi162.stdout: "recovery_state": [ 2024-09-17T16:42:37.245 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.245 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "Started/Primary/Active", 2024-09-17T16:42:37.245 INFO:tasks.workunit.client.0.smithi162.stdout: "enter_time": "2024-09-17T16:42:21.912949+0000", 2024-09-17T16:42:37.245 INFO:tasks.workunit.client.0.smithi162.stdout: "might_have_unfound": [ 2024-09-17T16:42:37.245 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.246 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "0(3)", 2024-09-17T16:42:37.246 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "already probed" 2024-09-17T16:42:37.246 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.246 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.246 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "1(1)", 2024-09-17T16:42:37.246 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "already probed" 2024-09-17T16:42:37.246 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.247 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.247 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "2(4)", 2024-09-17T16:42:37.247 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "already probed" 2024-09-17T16:42:37.247 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.247 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.247 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": "4(2)", 2024-09-17T16:42:37.248 INFO:tasks.workunit.client.0.smithi162.stdout: "status": "already probed" 2024-09-17T16:42:37.248 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:42:37.248 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.248 INFO:tasks.workunit.client.0.smithi162.stdout: "recovery_progress": { 2024-09-17T16:42:37.248 INFO:tasks.workunit.client.0.smithi162.stdout: "backfill_targets": [], 2024-09-17T16:42:37.248 INFO:tasks.workunit.client.0.smithi162.stdout: "waiting_on_backfill": [], 2024-09-17T16:42:37.249 INFO:tasks.workunit.client.0.smithi162.stdout: "last_backfill_started": "MIN", 2024-09-17T16:42:37.249 INFO:tasks.workunit.client.0.smithi162.stdout: "backfill_info": { 2024-09-17T16:42:37.249 INFO:tasks.workunit.client.0.smithi162.stdout: "begin": "MIN", 2024-09-17T16:42:37.249 INFO:tasks.workunit.client.0.smithi162.stdout: "end": "MIN", 2024-09-17T16:42:37.249 INFO:tasks.workunit.client.0.smithi162.stdout: "objects": [] 2024-09-17T16:42:37.249 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.249 INFO:tasks.workunit.client.0.smithi162.stdout: "peer_backfill_info": [], 2024-09-17T16:42:37.249 INFO:tasks.workunit.client.0.smithi162.stdout: "backfills_in_flight": [], 2024-09-17T16:42:37.249 INFO:tasks.workunit.client.0.smithi162.stdout: "recovering": [], 2024-09-17T16:42:37.249 INFO:tasks.workunit.client.0.smithi162.stdout: "pg_backend": { 2024-09-17T16:42:37.249 INFO:tasks.workunit.client.0.smithi162.stdout: "recovery_ops": [], 2024-09-17T16:42:37.249 INFO:tasks.workunit.client.0.smithi162.stdout: "read_ops": [] 2024-09-17T16:42:37.249 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:42:37.249 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:42:37.250 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.250 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-09-17T16:42:37.250 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "Started", 2024-09-17T16:42:37.250 INFO:tasks.workunit.client.0.smithi162.stdout: "enter_time": "2024-09-17T16:42:21.110475+0000" 2024-09-17T16:42:37.250 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-09-17T16:42:37.250 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-09-17T16:42:37.250 INFO:tasks.workunit.client.0.smithi162.stdout: "scrubber": { 2024-09-17T16:42:37.250 INFO:tasks.workunit.client.0.smithi162.stdout: "active": false, 2024-09-17T16:42:37.251 INFO:tasks.workunit.client.0.smithi162.stdout: "must_scrub": false, 2024-09-17T16:42:37.251 INFO:tasks.workunit.client.0.smithi162.stdout: "must_deep_scrub": false, 2024-09-17T16:42:37.251 INFO:tasks.workunit.client.0.smithi162.stdout: "must_repair": false, 2024-09-17T16:42:37.251 INFO:tasks.workunit.client.0.smithi162.stdout: "need_auto": false, 2024-09-17T16:42:37.251 INFO:tasks.workunit.client.0.smithi162.stdout: "scrub_reg_stamp": "2024-09-19T00:17:29.546814+0000", 2024-09-17T16:42:37.251 INFO:tasks.workunit.client.0.smithi162.stdout: "schedule": "not registered for scrubbing" 2024-09-17T16:42:37.251 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-09-17T16:42:37.252 INFO:tasks.workunit.client.0.smithi162.stdout: "agent_state": {} 2024-09-17T16:42:37.252 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-09-17T16:42:37.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-17T16:42:37.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: grep -q obj75 2024-09-17T16:42:37.287 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-17T16:42:37.287 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: jq .available_might_have_unfound 2024-09-17T16:42:37.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: check=true 2024-09-17T16:42:37.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: test true == true 2024-09-17T16:42:37.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:689: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-17T16:42:37.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:689: TEST_ec_recovery_unfound: jq '.might_have_unfound | length' 2024-09-17T16:42:37.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:689: TEST_ec_recovery_unfound: check=0 2024-09-17T16:42:37.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:690: TEST_ec_recovery_unfound: test 0 == 0 2024-09-17T16:42:37.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:693: TEST_ec_recovery_unfound: timeout 5 rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-17T16:42:42.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:694: TEST_ec_recovery_unfound: test 124 = 124 2024-09-17T16:42:42.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:696: TEST_ec_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-17T16:42:42.769 INFO:tasks.workunit.client.0.smithi162.stderr:pg has no unfound objects 2024-09-17T16:42:42.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:698: TEST_ec_recovery_unfound: wait_for_clean 2024-09-17T16:42:42.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:42:42.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:42:42.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:42:42.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:42:42.780 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:42:42.781 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:42:42.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:42:42.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:42:42.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:42:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:42:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:42:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:42:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:42:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:42:42.872 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:42:43.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:42:43.291 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:42:43.291 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:42:43.291 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:42:43.291 INFO:tasks.workunit.client.0.smithi162.stderr:4' 2024-09-17T16:42:43.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:42:43.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:42:43.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:42:43.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803826 2024-09-17T16:42:43.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803826 2024-09-17T16:42:43.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803826' 2024-09-17T16:42:43.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:42:43.447 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:42:43.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574895 2024-09-17T16:42:43.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574895 2024-09-17T16:42:43.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803826 1-55834574895' 2024-09-17T16:42:43.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:42:43.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:42:43.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201287 2024-09-17T16:42:43.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201287 2024-09-17T16:42:43.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803826 1-55834574895 2-236223201287' 2024-09-17T16:42:43.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:42:43.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:42:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018910 2024-09-17T16:42:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018910 2024-09-17T16:42:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803826 1-55834574895 2-236223201287 3-128849018910' 2024-09-17T16:42:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:42:43.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:42:43.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659156 2024-09-17T16:42:43.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659156 2024-09-17T16:42:43.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803826 1-55834574895 2-236223201287 3-128849018910 4-176093659156' 2024-09-17T16:42:43.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:42:43.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803826 2024-09-17T16:42:43.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:42:43.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:42:43.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803826 2024-09-17T16:42:43.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:42:43.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803826 2024-09-17T16:42:43.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803826' 2024-09-17T16:42:43.974 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803826 2024-09-17T16:42:43.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:42:44.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803824 -lt 25769803826 2024-09-17T16:42:44.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:42:45.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:42:45.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:42:45.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803827 -lt 25769803826 2024-09-17T16:42:45.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:42:45.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574895 2024-09-17T16:42:45.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:42:45.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:42:45.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574895 2024-09-17T16:42:45.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:42:45.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574895 2024-09-17T16:42:45.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574895' 2024-09-17T16:42:45.832 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574895 2024-09-17T16:42:45.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:42:46.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574896 -lt 55834574895 2024-09-17T16:42:46.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:42:46.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-236223201287 2024-09-17T16:42:46.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:42:46.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:42:46.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-236223201287 2024-09-17T16:42:46.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:42:46.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201287 2024-09-17T16:42:46.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 236223201287' 2024-09-17T16:42:46.232 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 236223201287 2024-09-17T16:42:46.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:42:46.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201287 -lt 236223201287 2024-09-17T16:42:46.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:42:46.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018910 2024-09-17T16:42:46.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:42:46.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:42:46.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018910 2024-09-17T16:42:46.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:42:46.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018910 2024-09-17T16:42:46.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018910' 2024-09-17T16:42:46.622 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 128849018910 2024-09-17T16:42:46.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:42:47.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018910 -lt 128849018910 2024-09-17T16:42:47.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:42:47.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-176093659156 2024-09-17T16:42:47.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:42:47.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:42:47.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-176093659156 2024-09-17T16:42:47.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:42:47.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659156 2024-09-17T16:42:47.052 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 176093659156 2024-09-17T16:42:47.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 176093659156' 2024-09-17T16:42:47.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:42:47.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659157 -lt 176093659156 2024-09-17T16:42:47.451 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:42:47.451 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:42:47.451 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:42:47.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:42:47.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:42:47.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:42:47.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:42:47.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:42:47.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:42:47.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:42:47.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:42:48.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:42:48.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:42:48.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:42:48.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:42:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:42:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:42:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:42:48.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: seq 1 100 2024-09-17T16:42:48.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:48.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj1 = obj75 ']' 2024-09-17T16:42:48.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-09-17T16:42:48.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:48.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:48.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj2 = obj75 ']' 2024-09-17T16:42:48.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-09-17T16:42:48.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:48.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:48.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj3 = obj75 ']' 2024-09-17T16:42:48.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj4 = obj75 ']' 2024-09-17T16:42:49.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj5 = obj75 ']' 2024-09-17T16:42:49.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj6 = obj75 ']' 2024-09-17T16:42:49.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj7 = obj75 ']' 2024-09-17T16:42:49.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj8 = obj75 ']' 2024-09-17T16:42:49.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj9 = obj75 ']' 2024-09-17T16:42:49.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj10 = obj75 ']' 2024-09-17T16:42:49.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj11 = obj75 ']' 2024-09-17T16:42:49.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj12 = obj75 ']' 2024-09-17T16:42:49.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj13 = obj75 ']' 2024-09-17T16:42:49.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj14 = obj75 ']' 2024-09-17T16:42:49.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj15 = obj75 ']' 2024-09-17T16:42:49.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj16 = obj75 ']' 2024-09-17T16:42:49.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj17 = obj75 ']' 2024-09-17T16:42:49.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj18 = obj75 ']' 2024-09-17T16:42:49.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj19 = obj75 ']' 2024-09-17T16:42:49.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj20 = obj75 ']' 2024-09-17T16:42:49.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj21 = obj75 ']' 2024-09-17T16:42:49.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj22 = obj75 ']' 2024-09-17T16:42:49.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj23 = obj75 ']' 2024-09-17T16:42:49.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj24 = obj75 ']' 2024-09-17T16:42:49.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj25 = obj75 ']' 2024-09-17T16:42:49.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj26 = obj75 ']' 2024-09-17T16:42:49.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj27 = obj75 ']' 2024-09-17T16:42:49.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj28 = obj75 ']' 2024-09-17T16:42:49.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj29 = obj75 ']' 2024-09-17T16:42:49.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj30 = obj75 ']' 2024-09-17T16:42:49.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-09-17T16:42:49.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:49.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:49.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj31 = obj75 ']' 2024-09-17T16:42:49.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj32 = obj75 ']' 2024-09-17T16:42:50.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj33 = obj75 ']' 2024-09-17T16:42:50.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj34 = obj75 ']' 2024-09-17T16:42:50.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj35 = obj75 ']' 2024-09-17T16:42:50.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj36 = obj75 ']' 2024-09-17T16:42:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj37 = obj75 ']' 2024-09-17T16:42:50.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj38 = obj75 ']' 2024-09-17T16:42:50.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj39 = obj75 ']' 2024-09-17T16:42:50.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj40 = obj75 ']' 2024-09-17T16:42:50.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj41 = obj75 ']' 2024-09-17T16:42:50.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj42 = obj75 ']' 2024-09-17T16:42:50.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj43 = obj75 ']' 2024-09-17T16:42:50.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj44 = obj75 ']' 2024-09-17T16:42:50.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj45 = obj75 ']' 2024-09-17T16:42:50.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj46 = obj75 ']' 2024-09-17T16:42:50.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj47 = obj75 ']' 2024-09-17T16:42:50.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj48 = obj75 ']' 2024-09-17T16:42:50.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj49 = obj75 ']' 2024-09-17T16:42:50.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj50 = obj75 ']' 2024-09-17T16:42:50.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj51 = obj75 ']' 2024-09-17T16:42:50.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj52 = obj75 ']' 2024-09-17T16:42:50.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj53 = obj75 ']' 2024-09-17T16:42:50.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj54 = obj75 ']' 2024-09-17T16:42:50.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj55 = obj75 ']' 2024-09-17T16:42:50.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj56 = obj75 ']' 2024-09-17T16:42:50.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj57 = obj75 ']' 2024-09-17T16:42:50.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj58 = obj75 ']' 2024-09-17T16:42:50.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj59 = obj75 ']' 2024-09-17T16:42:50.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-09-17T16:42:50.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:50.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:50.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj60 = obj75 ']' 2024-09-17T16:42:50.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj61 = obj75 ']' 2024-09-17T16:42:51.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj62 = obj75 ']' 2024-09-17T16:42:51.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj63 = obj75 ']' 2024-09-17T16:42:51.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj64 = obj75 ']' 2024-09-17T16:42:51.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj65 = obj75 ']' 2024-09-17T16:42:51.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj66 = obj75 ']' 2024-09-17T16:42:51.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj67 = obj75 ']' 2024-09-17T16:42:51.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj68 = obj75 ']' 2024-09-17T16:42:51.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj69 = obj75 ']' 2024-09-17T16:42:51.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj70 = obj75 ']' 2024-09-17T16:42:51.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj71 = obj75 ']' 2024-09-17T16:42:51.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj72 = obj75 ']' 2024-09-17T16:42:51.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj73 = obj75 ']' 2024-09-17T16:42:51.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj74 = obj75 ']' 2024-09-17T16:42:51.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj75 = obj75 ']' 2024-09-17T16:42:51.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:704: TEST_ec_recovery_unfound: rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.496 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj75: (2) No such file or directory 2024-09-17T16:42:51.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj76 = obj75 ']' 2024-09-17T16:42:51.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj77 = obj75 ']' 2024-09-17T16:42:51.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj78 = obj75 ']' 2024-09-17T16:42:51.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj79 = obj75 ']' 2024-09-17T16:42:51.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj80 = obj75 ']' 2024-09-17T16:42:51.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj81 = obj75 ']' 2024-09-17T16:42:51.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj82 = obj75 ']' 2024-09-17T16:42:51.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj83 = obj75 ']' 2024-09-17T16:42:51.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj84 = obj75 ']' 2024-09-17T16:42:51.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj85 = obj75 ']' 2024-09-17T16:42:51.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj86 = obj75 ']' 2024-09-17T16:42:51.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj87 = obj75 ']' 2024-09-17T16:42:51.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj88 = obj75 ']' 2024-09-17T16:42:51.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj89 = obj75 ']' 2024-09-17T16:42:51.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj90 = obj75 ']' 2024-09-17T16:42:51.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-09-17T16:42:51.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:51.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:51.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj91 = obj75 ']' 2024-09-17T16:42:51.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-09-17T16:42:52.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:52.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:52.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj92 = obj75 ']' 2024-09-17T16:42:52.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-09-17T16:42:52.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:52.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:52.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj93 = obj75 ']' 2024-09-17T16:42:52.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-09-17T16:42:52.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:52.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:52.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj94 = obj75 ']' 2024-09-17T16:42:52.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-09-17T16:42:52.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:52.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:52.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj95 = obj75 ']' 2024-09-17T16:42:52.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-09-17T16:42:52.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:52.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:52.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj96 = obj75 ']' 2024-09-17T16:42:52.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-09-17T16:42:52.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:52.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:52.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj97 = obj75 ']' 2024-09-17T16:42:52.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-09-17T16:42:52.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:52.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:52.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj98 = obj75 ']' 2024-09-17T16:42:52.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-09-17T16:42:52.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:52.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:52.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj99 = obj75 ']' 2024-09-17T16:42:52.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-09-17T16:42:52.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:52.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-17T16:42:52.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj100 = obj75 ']' 2024-09-17T16:42:52.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-09-17T16:42:52.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:52.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:711: TEST_ec_recovery_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-17T16:42:52.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:713: TEST_ec_recovery_unfound: delete_erasure_coded_pool pool-jerasure 2024-09-17T16:42:52.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:42:52.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-17T16:42:52.819 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-17T16:42:52.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-17T16:42:53.901 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-17T16:42:53.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-17T16:42:53.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:42:53.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:42:53.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:42:53.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:42:53.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:42:53.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:42:53.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:42:53.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:42:54.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:42:54.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:42:54.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:42:54.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:42:54.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:42:54.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:42:54.026 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:42:54.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:42:54.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:42:54.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:42:54.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:42:54.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:42:54.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:42:54.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:42:54.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:42:54.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:42:54.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:42:54.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:42:54.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:42:54.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:42:54.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-17T16:42:54.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-17T16:42:54.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-17T16:42:54.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-17T16:42:54.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:42:54.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:42:54.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:42:54.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:42:54.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:42:54.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:42:54.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:42:54.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:42:54.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:42:54.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:42:54.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:42:54.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:42:54.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:42:54.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:42:54.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:42:54.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:42:54.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:42:54.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:42:54.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:42:54.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:42:54.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:42:54.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:42:54.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:42:54.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:42:54.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:42:54.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:42:54.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:42:54.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:42:54.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-17T16:42:54.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:42:54.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:42:54.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:42:54.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.87940 2024-09-17T16:42:54.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:42:54.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T16:42:54.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:42:54.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-17T16:42:54.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-17T16:42:54.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-17T16:42:54.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:42:54.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:42:54.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:42:54.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-17T16:42:54.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-17T16:42:54.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:42:54.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:42:54.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:42:54.545 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:42:54.545 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:42:54.545 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:42:54.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:42:54.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T16:42:54.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:42:54.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:42:54.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:42:54.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:42:54.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:42:54.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:42:54.742 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:42:54.742 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:42:54.742 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:42:54.743 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:42:54.743 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:42:54.743 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:42:54.743 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:42:54.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:42:54.744 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get fsid 2024-09-17T16:42:54.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:42:54.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:42:54.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:42:54.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:42:54.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:42:54.822 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:42:54.822 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:42:54.822 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:42:54.823 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:42:54.823 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:42:54.823 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:42:54.823 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:42:54.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:42:54.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get mon_host 2024-09-17T16:42:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-17T16:42:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-17T16:42:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:42:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:42:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:42:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-17T16:42:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:42:55.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:42:55.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:42:55.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:42:55.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:42:55.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:42:55.751 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:42:55.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:42:55.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:42:55.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:42:55.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-17T16:42:55.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T16:42:56.062 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T16:42:56.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:42:57.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-17T16:42:57.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:42:57.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:42:57.075 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:42:57.075 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:42:57.075 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:42:57.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:42:57.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-17T16:42:57.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-mon.a.asok log flush 2024-09-17T16:42:57.147 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:42:57.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-17T16:42:57.156 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:42:54.576+0000 7fac3242cc80 0 load: jerasure load: lrc 2024-09-17T16:42:57.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_single_recovery_error td/test-erasure-eio 2024-09-17T16:42:57.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:410: TEST_ec_single_recovery_error: local dir=td/test-erasure-eio 2024-09-17T16:42:57.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:411: TEST_ec_single_recovery_error: local objname=myobject 2024-09-17T16:42:57.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:413: TEST_ec_single_recovery_error: setup_osds 7 2024-09-17T16:42:57.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-17T16:42:57.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-17T16:42:57.157 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-17T16:42:57.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-17T16:42:57.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:42:57.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-17T16:42:57.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:42:57.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:42:57.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:42:57.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:42:57.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-17T16:42:57.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:42:57.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:42:57.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:42:57.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:42:57.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-17T16:42:57.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-17T16:42:57.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:42:57.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:42:57.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:42:57.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:42:57.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:42:57.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:42:57.165 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:42:57.165 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:42:57.165 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:42:57.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:42:57.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:42:57.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:42:57.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:42:57.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:42:57.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:42:57.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:42:57.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:42:57.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:42:57.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:42:57.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:42:57.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:42:57.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:42:57.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-17T16:42:57.166 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:42:57.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b64892f9-7c45-45a4-9525-b37d97b82f1a 2024-09-17T16:42:57.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b64892f9-7c45-45a4-9525-b37d97b82f1a' 2024-09-17T16:42:57.167 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 b64892f9-7c45-45a4-9525-b37d97b82f1a 2024-09-17T16:42:57.167 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:42:57.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCRselmD0/IChAAloUmgwgkszeSxwGBjQUt+A== 2024-09-17T16:42:57.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCRselmD0/IChAAloUmgwgkszeSxwGBjQUt+A=="}' 2024-09-17T16:42:57.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b64892f9-7c45-45a4-9525-b37d97b82f1a -i td/test-erasure-eio/0/new.json 2024-09-17T16:42:57.446 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:42:57.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-17T16:42:57.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCRselmD0/IChAAloUmgwgkszeSxwGBjQUt+A== --osd-uuid b64892f9-7c45-45a4-9525-b37d97b82f1a 2024-09-17T16:42:57.486 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:42:57.483+0000 7f7af4ed6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:42:57.515 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:42:57.512+0000 7f7af4ed6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:42:57.516 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:42:57.513+0000 7f7af4ed6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:42:57.517 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:42:57.514+0000 7f7af4ed6640 -1 bdev(0x5603f3b99c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:42:57.517 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:42:57.514+0000 7f7af4ed6640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-17T16:43:03.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-17T16:43:03.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:43:03.465 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:43:03.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:43:03.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:43:04.001 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:43:04.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:43:04.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:43:04.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:43:04.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:43:04.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:43:04.028 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:43:04.026+0000 7f498060d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:43:04.047 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:43:04.045+0000 7f498060d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:43:04.048 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:43:04.046+0000 7f498060d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:43:04.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:43:04.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:43:04.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:43:04.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:43:04.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:43:04.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:04.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:43:04.424 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:43:04.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:04.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:04.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:04.900 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:43:04.898+0000 7f498060d640 -1 Falling back to public interface 2024-09-17T16:43:05.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:05.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:05.847 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:43:05.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:43:05.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:05.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:06.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:07.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:07.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:07.281 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:43:07.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:43:07.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:07.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:07.574 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:43:07.572+0000 7f498060d640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:43:07.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:08.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:08.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:08.678 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:43:08.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:43:08.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:08.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:09.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:10.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:10.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:10.077 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:43:10.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:43:10.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:10.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:10.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:11.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:11.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:11.506 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:43:11.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:43:11.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:11.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:11.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:12.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:12.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:12.908 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:43:12.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:43:12.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:12.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:13.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:14.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:14.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:14.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:43:14.425 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:43:14.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:14.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:14.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:15.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:15.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:15.853 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:43:15.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:43:15.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:15.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:16.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:17.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:17.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:17.453 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:43:17.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:43:17.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:17.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:17.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:18.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:18.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:18.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:43:18.873 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:43:18.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:18.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:19.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:20.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:20.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:20.334 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:43:20.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:43:20.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:20.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:21.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:22.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:22.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:22.163 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:43:22.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:43:22.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:22.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:23.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:24.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:24.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:24.168 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:43:24.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:43:24.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:24.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:24.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:25.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:25.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:25.703 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:43:25.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:43:25.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:25.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:26.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:27.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:27.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:27.221 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:43:27.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:43:27.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:27.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:27.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:28.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:28.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:28.964 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:43:28.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:43:28.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:28.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:29.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:30.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:30.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:30.852 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:43:30.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:43:30.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:30.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:43:31.322 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/601243653,v1:127.0.0.1:6803/601243653] [v2:127.0.0.1:6804/601243653,v1:127.0.0.1:6805/601243653] exists,up b64892f9-7c45-45a4-9525-b37d97b82f1a 2024-09-17T16:43:31.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:43:31.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:43:31.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:43:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:43:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-17T16:43:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:43:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:43:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T16:43:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:43:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:43:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:43:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:43:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:43:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:43:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:43:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:43:31.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:43:31.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:43:31.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:43:31.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:43:31.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:43:31.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:43:31.327 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:43:31.327 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:43:31.327 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:43:31.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:43:31.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:43:31.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:43:31.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:43:31.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:43:31.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:43:31.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:43:31.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:43:31.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:43:31.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:43:31.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:43:31.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:43:31.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:43:31.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:43:31.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:43:31.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2c5c5eee-6601-4d0f-9dc7-6daebd42bb02 2024-09-17T16:43:31.329 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 2c5c5eee-6601-4d0f-9dc7-6daebd42bb02 2024-09-17T16:43:31.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 2c5c5eee-6601-4d0f-9dc7-6daebd42bb02' 2024-09-17T16:43:31.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:43:31.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCzselmMK1jFBAAbugw7UWPl9uLH642PWBPoA== 2024-09-17T16:43:31.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCzselmMK1jFBAAbugw7UWPl9uLH642PWBPoA=="}' 2024-09-17T16:43:31.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2c5c5eee-6601-4d0f-9dc7-6daebd42bb02 -i td/test-erasure-eio/1/new.json 2024-09-17T16:43:31.814 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:43:31.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-17T16:43:31.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCzselmMK1jFBAAbugw7UWPl9uLH642PWBPoA== --osd-uuid 2c5c5eee-6601-4d0f-9dc7-6daebd42bb02 2024-09-17T16:43:31.855 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:43:31.853+0000 7f0d5de14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:43:31.874 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:43:31.872+0000 7f0d5de14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:43:31.875 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:43:31.873+0000 7f0d5de14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:43:31.876 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:43:31.874+0000 7f0d5de14640 -1 bdev(0x55f71133fc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-17T16:43:31.876 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:43:31.874+0000 7f0d5de14640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-17T16:43:37.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-17T16:43:37.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:43:37.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T16:43:37.912 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T16:43:37.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:43:38.450 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T16:43:38.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T16:43:38.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:43:38.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:43:38.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:43:38.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:43:38.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:43:38.475+0000 7f615521e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:43:38.494 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:43:38.491+0000 7f615521e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:43:38.495 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:43:38.492+0000 7f615521e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:43:38.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T16:43:38.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:43:38.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:43:38.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:43:38.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:43:38.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:38.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:43:38.909 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:43:38.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:38.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:43:39.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:39.611 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:43:39.608+0000 7f615521e640 -1 Falling back to public interface 2024-09-17T16:43:40.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:40.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:40.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:43:40.344 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:43:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:43:40.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:41.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:41.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:41.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:43:41.770 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:43:41.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:41.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:43:42.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:42.789 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:43:42.786+0000 7f615521e640 -1 osd.1 0 log_to_monitors true 2024-09-17T16:43:43.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:43.180 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:43:43.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:43.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:43:43.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:43.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:43:43.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:44.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:44.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:44.637 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:43:44.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:43:44.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:44.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:43:45.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:46.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:46.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:46.129 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:43:46.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:43:46.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:46.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:43:46.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:47.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:47.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:47.660 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:43:47.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:43:47.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:47.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:43:48.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:49.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:49.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:49.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:43:49.157 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:43:49.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:49.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:43:49.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:50.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:50.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:50.676 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:43:50.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:43:50.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:50.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:43:51.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:52.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:52.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:52.131 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:43:52.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:43:52.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:52.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:43:52.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:53.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:53.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:53.688 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:43:53.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:43:53.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:53.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:43:54.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:55.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:55.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:55.142 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:43:55.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:43:55.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:55.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:43:55.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:56.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:56.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:56.600 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:43:56.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:43:56.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:56.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:43:57.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:43:58.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:43:58.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:43:58.444 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:43:58.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:43:58.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:43:58.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:43:59.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:00.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:00.238 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:44:00.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:00.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:44:00.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:00.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:44:01.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:02.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:02.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:02.157 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:44:02.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:44:02.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:02.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:44:03.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:04.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:04.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:04.098 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:44:04.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:44:04.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:04.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:44:04.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:05.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:05.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:05.886 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:44:05.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:44:05.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:05.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:44:06.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:07.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:07.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:07.989 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:44:07.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:44:07.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:07.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:44:08.489 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/855858164,v1:127.0.0.1:6811/855858164] [v2:127.0.0.1:6812/855858164,v1:127.0.0.1:6813/855858164] exists,up 2c5c5eee-6601-4d0f-9dc7-6daebd42bb02 2024-09-17T16:44:08.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:44:08.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:44:08.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:44:08.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:44:08.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-17T16:44:08.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:44:08.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:44:08.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T16:44:08.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:44:08.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:44:08.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:44:08.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:44:08.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:44:08.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:44:08.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:44:08.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:44:08.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:44:08.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:44:08.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:44:08.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:44:08.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:44:08.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:44:08.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:44:08.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:44:08.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:44:08.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7d2f0774-ad13-4e24-ba8e-cccd34a9aa34 2024-09-17T16:44:08.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 7d2f0774-ad13-4e24-ba8e-cccd34a9aa34' 2024-09-17T16:44:08.496 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 7d2f0774-ad13-4e24-ba8e-cccd34a9aa34 2024-09-17T16:44:08.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:44:08.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDYselmkLtdHhAA5H9i75InxWwmhG2kVuXJYA== 2024-09-17T16:44:08.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDYselmkLtdHhAA5H9i75InxWwmhG2kVuXJYA=="}' 2024-09-17T16:44:08.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7d2f0774-ad13-4e24-ba8e-cccd34a9aa34 -i td/test-erasure-eio/2/new.json 2024-09-17T16:44:08.961 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:44:08.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-17T16:44:08.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDYselmkLtdHhAA5H9i75InxWwmhG2kVuXJYA== --osd-uuid 7d2f0774-ad13-4e24-ba8e-cccd34a9aa34 2024-09-17T16:44:09.004 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:09.002+0000 7f57c9c31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:44:09.014 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:09.012+0000 7f57c9c31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:44:09.015 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:09.013+0000 7f57c9c31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:44:09.015 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:09.013+0000 7f57c9c31640 -1 bdev(0x556272c3fc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-17T16:44:09.015 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:09.013+0000 7f57c9c31640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-17T16:44:15.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-17T16:44:15.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:44:15.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T16:44:15.577 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T16:44:15.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:44:16.140 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T16:44:16.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T16:44:16.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:44:16.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:44:16.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:44:16.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:44:16.167 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:16.165+0000 7fb0d14e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:44:16.176 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:16.174+0000 7fb0d14e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:44:16.177 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:16.175+0000 7fb0d14e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:44:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T16:44:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:44:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:44:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:44:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:44:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:44:16.592 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:44:16.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:16.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:17.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:17.539 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:17.537+0000 7fb0d14e8640 -1 Falling back to public interface 2024-09-17T16:44:18.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:18.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:18.009 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:44:18.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:44:18.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:18.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:18.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:19.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:19.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:19.444 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:44:19.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:44:19.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:19.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:19.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:20.422 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:20.420+0000 7fb0d14e8640 -1 osd.2 0 log_to_monitors true 2024-09-17T16:44:20.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:20.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:20.841 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:44:20.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:44:20.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:20.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:21.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:22.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:22.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:22.238 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:44:22.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:44:22.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:22.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:22.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:23.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:23.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:23.635 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:44:23.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:44:23.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:23.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:24.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:25.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:25.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:25.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:44:25.034 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:44:25.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:25.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:25.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:26.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:26.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:26.497 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:44:26.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:44:26.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:26.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:26.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:27.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:27.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:27.891 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:44:27.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:44:27.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:27.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:28.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:29.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:29.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:29.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:44:29.392 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:44:29.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:29.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:29.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:30.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:30.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:30.813 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:44:30.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:44:30.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:30.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:31.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:32.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:32.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:32.253 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:44:32.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:44:32.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:32.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:33.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:33.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:33.652 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:44:33.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:44:33.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:33.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:34.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:35.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:35.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:35.233 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:44:35.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:44:35.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:35.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:36.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:37.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:37.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:37.029 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:44:37.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:44:37.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:37.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:37.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:38.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:38.790 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:44:38.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:38.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:44:38.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:38.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:39.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:40.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:40.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:40.626 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:44:40.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:44:40.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:40.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:41.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:42.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:42.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:42.543 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:44:42.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:44:42.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:42.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:43.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:44.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:44.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:44.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:44:44.671 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:44:44.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:44.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:44:45.170 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3800156587,v1:127.0.0.1:6819/3800156587] [v2:127.0.0.1:6820/3800156587,v1:127.0.0.1:6821/3800156587] exists,up 7d2f0774-ad13-4e24-ba8e-cccd34a9aa34 2024-09-17T16:44:45.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:44:45.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:44:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:44:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:44:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-17T16:44:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:44:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:44:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T16:44:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:44:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T16:44:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:44:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:44:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:44:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:44:45.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T16:44:45.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T16:44:45.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:44:45.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:44:45.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:44:45.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:44:45.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:44:45.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:44:45.176 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:44:45.176 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:44:45.176 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:44:45.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:44:45.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:44:45.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:44:45.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:44:45.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:44:45.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:44:45.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:44:45.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:44:45.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:44:45.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:44:45.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:44:45.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:44:45.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:44:45.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T16:44:45.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:44:45.177 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 b6042083-a7fe-42b9-9cc1-be65a434d8c8 2024-09-17T16:44:45.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b6042083-a7fe-42b9-9cc1-be65a434d8c8 2024-09-17T16:44:45.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 b6042083-a7fe-42b9-9cc1-be65a434d8c8' 2024-09-17T16:44:45.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:44:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD9selmYpZgCxAAPMMZtfjtp2xCH0r3/cZxrw== 2024-09-17T16:44:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD9selmYpZgCxAAPMMZtfjtp2xCH0r3/cZxrw=="}' 2024-09-17T16:44:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b6042083-a7fe-42b9-9cc1-be65a434d8c8 -i td/test-erasure-eio/3/new.json 2024-09-17T16:44:45.999 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:44:46.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-17T16:44:46.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD9selmYpZgCxAAPMMZtfjtp2xCH0r3/cZxrw== --osd-uuid b6042083-a7fe-42b9-9cc1-be65a434d8c8 2024-09-17T16:44:46.041 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:46.039+0000 7f4331da8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:44:46.043 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:46.041+0000 7f4331da8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:44:46.044 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:46.042+0000 7f4331da8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:44:46.044 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:46.042+0000 7f4331da8640 -1 bdev(0x55d552201c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-17T16:44:46.044 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:46.042+0000 7f4331da8640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-17T16:44:53.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-17T16:44:53.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:44:53.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T16:44:53.040 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-17T16:44:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:44:53.579 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T16:44:53.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T16:44:53.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:44:53.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:44:53.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:44:53.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:44:53.606 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:53.604+0000 7f55565fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:44:53.619 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:53.617+0000 7f55565fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:44:53.620 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:53.618+0000 7f55565fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:44:54.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T16:44:54.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:44:54.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:44:54.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:44:54.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:44:54.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:54.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:44:54.016 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:44:54.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:54.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:44:54.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:54.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:54.461+0000 7f55565fd640 -1 Falling back to public interface 2024-09-17T16:44:55.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:55.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:55.400 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:44:55.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:44:55.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:55.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:44:55.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:56.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:56.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:56.830 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:44:56.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:44:56.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:56.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:44:56.831 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:44:56.829+0000 7f55565fd640 -1 osd.3 0 log_to_monitors true 2024-09-17T16:44:57.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:44:58.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:44:58.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:44:58.350 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:44:58.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:44:58.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:44:58.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:44:59.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:00.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:00.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:00.041 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:45:00.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:45:00.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:00.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:45:00.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:01.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:01.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:01.481 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:45:01.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:45:01.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:01.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:45:02.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:03.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:03.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:03.036 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:45:03.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:45:03.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:03.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:45:03.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:04.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:04.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:04.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:45:04.497 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:45:04.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:04.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:45:05.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:06.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:06.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:06.158 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:45:06.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:45:06.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:06.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:45:06.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:07.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:07.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:07.693 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:45:07.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:45:07.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:07.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:45:08.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:09.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:09.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:09.170 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:45:09.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:45:09.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:09.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:45:09.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:10.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:10.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:10.701 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:45:10.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:45:10.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:10.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:45:11.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:12.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:12.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:12.840 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:45:12.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:45:12.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:12.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:45:13.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:14.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:14.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:14.323 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:45:14.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:45:14.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:14.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:45:15.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:16.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:16.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:16.302 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:45:16.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:45:16.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:16.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:45:17.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:18.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:18.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:18.196 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:45:18.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:45:18.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:18.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:45:19.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:20.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:20.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:20.241 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:45:20.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:45:20.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:20.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:45:20.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:21.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:21.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:21.957 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:45:21.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:45:21.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:21.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:45:22.536 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3948833370,v1:127.0.0.1:6827/3948833370] [v2:127.0.0.1:6828/3948833370,v1:127.0.0.1:6829/3948833370] exists,up b6042083-a7fe-42b9-9cc1-be65a434d8c8 2024-09-17T16:45:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:45:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:45:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:45:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:45:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-17T16:45:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:45:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:45:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-17T16:45:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:45:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-17T16:45:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:45:22.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:45:22.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:45:22.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:45:22.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-17T16:45:22.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-17T16:45:22.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:45:22.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:45:22.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:45:22.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:45:22.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:45:22.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:45:22.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:45:22.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:45:22.542 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:45:22.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:45:22.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:45:22.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:45:22.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:45:22.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:45:22.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:45:22.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:45:22.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:45:22.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:45:22.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:45:22.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:45:22.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:45:22.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:45:22.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-17T16:45:22.543 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:45:22.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=14c2a946-77a6-4596-ac4f-4816a004b281 2024-09-17T16:45:22.543 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 14c2a946-77a6-4596-ac4f-4816a004b281 2024-09-17T16:45:22.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 14c2a946-77a6-4596-ac4f-4816a004b281' 2024-09-17T16:45:22.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:45:22.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAisulm2M8rIRAA0in/atD3f1fvzXm27tBPYw== 2024-09-17T16:45:22.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAisulm2M8rIRAA0in/atD3f1fvzXm27tBPYw=="}' 2024-09-17T16:45:22.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 14c2a946-77a6-4596-ac4f-4816a004b281 -i td/test-erasure-eio/4/new.json 2024-09-17T16:45:23.396 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:45:23.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-17T16:45:23.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAisulm2M8rIRAA0in/atD3f1fvzXm27tBPYw== --osd-uuid 14c2a946-77a6-4596-ac4f-4816a004b281 2024-09-17T16:45:23.437 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:45:23.434+0000 7fea71cb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:45:23.439 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:45:23.436+0000 7fea71cb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:45:23.440 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:45:23.437+0000 7fea71cb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:45:23.440 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:45:23.437+0000 7fea71cb6640 -1 bdev(0x55daf7431c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-17T16:45:23.441 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:45:23.438+0000 7fea71cb6640 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-17T16:45:30.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-17T16:45:30.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:45:30.296 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-09-17T16:45:30.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-17T16:45:30.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:45:30.835 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-09-17T16:45:30.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-17T16:45:30.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:45:30.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:45:30.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:45:30.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:45:30.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:45:30.860+0000 7f9631a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:45:30.877 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:45:30.875+0000 7f9631a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:45:30.878 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:45:30.876+0000 7f9631a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:45:31.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-17T16:45:31.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:45:31.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-17T16:45:31.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:45:31.283 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:45:31.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:45:31.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:31.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:45:31.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:31.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:31.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:31.742 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:45:31.740+0000 7f9631a18640 -1 Falling back to public interface 2024-09-17T16:45:32.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:32.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:32.710 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:45:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:45:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:33.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:34.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:34.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:34.121 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:45:34.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:45:34.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:34.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:34.489 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:45:34.487+0000 7f9631a18640 -1 osd.4 0 log_to_monitors true 2024-09-17T16:45:34.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:35.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:35.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:35.525 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:45:35.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:45:35.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:35.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:36.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:37.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:37.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:37.039 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:45:37.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:45:37.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:37.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:37.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:38.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:38.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:38.808 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:45:38.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:45:38.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:38.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:39.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:40.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:40.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:40.416 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:45:40.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:45:40.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:40.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:40.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:41.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:41.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:41.819 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:45:41.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:45:41.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:41.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:42.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:43.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:43.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:43.225 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:45:43.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:45:43.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:43.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:43.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:44.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:44.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:44.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:45:44.681 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:45:44.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:44.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:45.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:46.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:46.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:46.147 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:45:46.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:45:46.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:46.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:46.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:47.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:47.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:47.552 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:45:47.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:45:47.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:47.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:47.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:48.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:48.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:48.948 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:45:48.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:45:48.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:48.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:49.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:50.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:50.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:50.723 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:45:50.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:45:50.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:50.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:51.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:52.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:52.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:52.190 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:45:52.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:45:52.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:52.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:53.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:54.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:54.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:54.429 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:45:54.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:45:54.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:54.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:54.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:55.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:55.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:55.929 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:45:55.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:45:55.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:55.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:56.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:57.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:57.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:57.440 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:45:57.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:45:57.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:57.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:58.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:45:59.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:45:59.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:45:59.134 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:45:59.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:45:59.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:45:59.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:45:59.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:00.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:00.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:00.770 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-09-17T16:46:00.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T16:46:00.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:00.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:46:01.278 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 42 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/4293627753,v1:127.0.0.1:6835/4293627753] [v2:127.0.0.1:6836/4293627753,v1:127.0.0.1:6837/4293627753] exists,up 14c2a946-77a6-4596-ac4f-4816a004b281 2024-09-17T16:46:01.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:46:01.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:46:01.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:46:01.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:46:01.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-17T16:46:01.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:46:01.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:46:01.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-17T16:46:01.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:46:01.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-17T16:46:01.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:46:01.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:46:01.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:46:01.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:46:01.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-17T16:46:01.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-17T16:46:01.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:46:01.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:46:01.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:46:01.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:46:01.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:46:01.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:46:01.283 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:46:01.283 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:46:01.283 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:46:01.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:46:01.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:46:01.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:46:01.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:46:01.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:46:01.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:46:01.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:46:01.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:46:01.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:46:01.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:46:01.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:46:01.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:46:01.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:46:01.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-17T16:46:01.285 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:46:01.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=082f7fc7-7b59-4ebb-bd01-7cd9149d6a13 2024-09-17T16:46:01.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 082f7fc7-7b59-4ebb-bd01-7cd9149d6a13' 2024-09-17T16:46:01.286 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 082f7fc7-7b59-4ebb-bd01-7cd9149d6a13 2024-09-17T16:46:01.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:46:01.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBJsulmmj3XERAAbM5TDGgPrYmCgjfYug4kyw== 2024-09-17T16:46:01.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBJsulmmj3XERAAbM5TDGgPrYmCgjfYug4kyw=="}' 2024-09-17T16:46:01.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 082f7fc7-7b59-4ebb-bd01-7cd9149d6a13 -i td/test-erasure-eio/5/new.json 2024-09-17T16:46:02.091 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:46:02.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-17T16:46:02.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJsulmmj3XERAAbM5TDGgPrYmCgjfYug4kyw== --osd-uuid 082f7fc7-7b59-4ebb-bd01-7cd9149d6a13 2024-09-17T16:46:02.132 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:02.129+0000 7fddd0d25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:46:02.133 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:02.131+0000 7fddd0d25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:46:02.135 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:02.132+0000 7fddd0d25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:46:02.135 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:02.132+0000 7fddd0d25640 -1 bdev(0x55d411089c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2024-09-17T16:46:02.135 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:02.132+0000 7fddd0d25640 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-17T16:46:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-17T16:46:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:46:08.638 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2024-09-17T16:46:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-17T16:46:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:46:09.184 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2024-09-17T16:46:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-17T16:46:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:46:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:46:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:46:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:46:09.213 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:09.209+0000 7f51fcd29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:46:09.214 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:09.212+0000 7f51fcd29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:46:09.215 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:09.213+0000 7f51fcd29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:46:09.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-17T16:46:09.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:46:09.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T16:46:09.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:46:09.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:46:09.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:09.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:46:09.612 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:46:09.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:09.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:10.053 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:10.050+0000 7f51fcd29640 -1 Falling back to public interface 2024-09-17T16:46:10.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:11.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:11.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:11.063 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:46:11.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:46:11.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:11.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:11.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:12.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:12.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:12.499 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:46:12.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:46:12.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:12.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:12.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:13.385 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:13.383+0000 7f51fcd29640 -1 osd.5 0 log_to_monitors true 2024-09-17T16:46:13.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:13.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:13.897 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:46:13.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:46:13.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:13.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:14.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:15.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:15.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:15.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:46:15.693 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:46:15.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:15.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:16.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:17.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:17.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:17.366 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:46:17.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:46:17.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:17.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:17.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:18.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:18.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:18.768 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:46:18.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:46:18.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:18.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:19.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:20.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:20.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:20.297 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:46:20.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:46:20.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:20.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:20.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:21.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:21.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:21.713 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:46:21.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:46:21.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:21.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:22.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:23.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:23.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:23.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:46:23.141 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:46:23.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:23.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:23.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:24.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:24.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:24.657 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:46:24.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:46:24.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:24.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:25.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:26.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:26.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:26.131 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:46:26.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:46:26.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:26.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:26.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:27.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:27.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:27.533 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:46:27.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:46:27.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:27.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:28.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:29.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:29.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:29.407 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:46:29.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:46:29.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:29.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:30.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:31.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:31.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:31.312 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:46:31.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:46:31.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:31.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:32.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:33.198 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:46:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:46:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:34.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:35.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:35.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:35.040 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:46:35.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:46:35.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:35.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:35.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:36.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:36.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:36.717 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:46:36.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:46:36.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:36.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:37.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:38.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:38.744 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:46:38.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:38.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:46:38.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:38.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:46:39.208 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 52 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3961312041,v1:127.0.0.1:6843/3961312041] [v2:127.0.0.1:6844/3961312041,v1:127.0.0.1:6845/3961312041] exists,up 082f7fc7-7b59-4ebb-bd01-7cd9149d6a13 2024-09-17T16:46:39.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:46:39.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:46:39.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:46:39.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:46:39.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-17T16:46:39.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:46:39.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:46:39.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-17T16:46:39.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:46:39.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-17T16:46:39.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:46:39.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:46:39.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:46:39.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:46:39.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-17T16:46:39.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-17T16:46:39.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:46:39.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:46:39.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:46:39.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:46:39.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:46:39.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:46:39.217 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:46:39.217 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:46:39.217 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:46:39.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:46:39.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:46:39.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:46:39.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:46:39.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:46:39.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:46:39.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:46:39.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:46:39.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:46:39.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:46:39.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:46:39.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:46:39.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:46:39.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-17T16:46:39.218 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:46:39.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1a0ab23d-d77e-4b94-bfc0-433b08845e85 2024-09-17T16:46:39.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 1a0ab23d-d77e-4b94-bfc0-433b08845e85' 2024-09-17T16:46:39.218 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 1a0ab23d-d77e-4b94-bfc0-433b08845e85 2024-09-17T16:46:39.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:46:39.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBvsulmkC6jDRAAH8WTR2YNcBh+QhMAJB17Tw== 2024-09-17T16:46:39.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBvsulmkC6jDRAAH8WTR2YNcBh+QhMAJB17Tw=="}' 2024-09-17T16:46:39.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1a0ab23d-d77e-4b94-bfc0-433b08845e85 -i td/test-erasure-eio/6/new.json 2024-09-17T16:46:39.790 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:46:39.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-17T16:46:39.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvsulmkC6jDRAAH8WTR2YNcBh+QhMAJB17Tw== --osd-uuid 1a0ab23d-d77e-4b94-bfc0-433b08845e85 2024-09-17T16:46:39.832 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:39.830+0000 7fdac140d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:46:39.834 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:39.832+0000 7fdac140d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:46:39.835 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:39.833+0000 7fdac140d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:46:39.835 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:39.833+0000 7fdac140d640 -1 bdev(0x55b6139efc00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2024-09-17T16:46:39.835 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:39.833+0000 7fdac140d640 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-17T16:46:46.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-17T16:46:46.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:46:46.958 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2024-09-17T16:46:46.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-17T16:46:46.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:46:47.498 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2024-09-17T16:46:47.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-17T16:46:47.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:46:47.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:46:47.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:46:47.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:46:47.525 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:47.522+0000 7f6ea11c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:46:47.536 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:47.533+0000 7f6ea11c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:46:47.537 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:47.534+0000 7f6ea11c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:46:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-17T16:46:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:46:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-17T16:46:47.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:46:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:46:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:46:47.914 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:46:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:47.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:46:48.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:48.898 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:48.895+0000 7f6ea11c5640 -1 Falling back to public interface 2024-09-17T16:46:49.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:49.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:49.337 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:46:49.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:46:49.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:49.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:46:49.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:50.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:50.771 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:46:50.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:50.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:46:50.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:50.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:46:51.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:51.809 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:46:51.806+0000 7f6ea11c5640 -1 osd.6 0 log_to_monitors true 2024-09-17T16:46:52.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:52.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:52.216 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:46:52.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:46:52.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:52.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:46:52.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:53.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:53.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:53.614 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:46:53.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:46:53.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:53.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:46:54.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:55.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:55.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:55.079 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:46:55.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:46:55.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:55.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:46:55.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:56.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:56.688 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:46:56.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:56.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:46:56.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:56.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:46:57.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:58.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:58.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:58.229 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:46:58.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:46:58.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:58.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:46:58.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:46:59.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:46:59.690 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:46:59.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:46:59.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:46:59.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:46:59.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:47:00.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:01.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:01.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:01.175 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:47:01.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:47:01.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:01.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:47:01.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:02.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:02.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:02.636 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:47:02.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:47:02.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:02.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:47:03.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:04.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:04.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:04.147 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:47:04.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:47:04.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:04.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:47:04.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:05.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:05.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:05.654 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:47:05.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:47:05.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:05.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:47:06.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:07.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:07.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:07.294 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:47:07.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:47:07.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:07.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:47:08.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:09.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:09.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:09.064 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:47:09.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:47:09.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:09.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:47:09.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:10.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:10.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:10.766 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:47:10.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:47:10.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:10.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:47:11.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:12.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:12.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:12.569 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:47:12.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:47:12.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:12.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:47:13.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:14.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:14.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:14.517 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:47:14.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:47:14.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:14.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:47:15.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:16.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:16.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:16.604 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:47:16.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:47:16.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:16.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:47:17.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:18.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:18.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:18.006 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-09-17T16:47:18.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T16:47:18.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:18.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:47:18.559 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3294830962,v1:127.0.0.1:6851/3294830962] [v2:127.0.0.1:6852/3294830962,v1:127.0.0.1:6853/3294830962] exists,up 1a0ab23d-d77e-4b94-bfc0-433b08845e85 2024-09-17T16:47:18.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:47:18.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:47:18.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:47:18.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-17T16:47:18.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:47:18.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:47:18.561 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:47:18.561 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:47:18.561 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:47:18.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T16:47:18.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-17T16:47:18.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.0.asok log flush 2024-09-17T16:47:18.641 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:47:18.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-17T16:47:18.653 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:43:04.902+0000 7f498060d640 0 load: jerasure load: lrc 2024-09-17T16:47:18.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:415: TEST_ec_single_recovery_error: local poolname=pool-jerasure 2024-09-17T16:47:18.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:416: TEST_ec_single_recovery_error: create_erasure_coded_pool pool-jerasure 3 2 2024-09-17T16:47:18.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:47:18.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-17T16:47:18.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-17T16:47:18.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-17T16:47:18.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-17T16:47:18.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-17T16:47:18.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-17T16:47:19.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-17T16:47:19.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-17T16:47:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-17T16:47:20.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:47:21.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-17T16:47:21.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:47:21.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:47:21.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:47:21.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:47:21.465 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:47:21.465 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:47:21.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:47:21.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:47:21.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:47:21.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:47:21.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:47:21.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:47:21.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:47:21.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:47:21.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:47:22.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:47:22.087 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:47:22.087 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:47:22.087 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:47:22.087 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:47:22.087 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:47:22.087 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-17T16:47:22.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:47:22.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:47:22.087 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:47:22.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803831 2024-09-17T16:47:22.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803831 2024-09-17T16:47:22.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831' 2024-09-17T16:47:22.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:47:22.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:47:22.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574894 2024-09-17T16:47:22.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574894 2024-09-17T16:47:22.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574894' 2024-09-17T16:47:22.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:47:22.440 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:47:22.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345960 2024-09-17T16:47:22.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345960 2024-09-17T16:47:22.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574894 2-85899345960' 2024-09-17T16:47:22.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:47:22.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:47:22.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018908 2024-09-17T16:47:22.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018908 2024-09-17T16:47:22.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574894 2-85899345960 3-128849018908' 2024-09-17T16:47:22.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:47:22.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:47:22.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=180388626453 2024-09-17T16:47:22.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 180388626453 2024-09-17T16:47:22.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574894 2-85899345960 3-128849018908 4-180388626453' 2024-09-17T16:47:22.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:47:22.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:47:22.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=223338299412 2024-09-17T16:47:22.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 223338299412 2024-09-17T16:47:22.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574894 2-85899345960 3-128849018908 4-180388626453 5-223338299412' 2024-09-17T16:47:22.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:47:22.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:47:23.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=261993005063 2024-09-17T16:47:23.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 261993005063 2024-09-17T16:47:23.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574894 2-85899345960 3-128849018908 4-180388626453 5-223338299412 6-261993005063' 2024-09-17T16:47:23.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:47:23.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803831 2024-09-17T16:47:23.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:47:23.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:47:23.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803831 2024-09-17T16:47:23.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:47:23.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803831 2024-09-17T16:47:23.012 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803831 2024-09-17T16:47:23.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803831' 2024-09-17T16:47:23.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:47:23.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803829 -lt 25769803831 2024-09-17T16:47:23.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:47:24.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:47:24.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:47:24.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803831 -lt 25769803831 2024-09-17T16:47:24.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:47:24.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574894 2024-09-17T16:47:24.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:47:24.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:47:24.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574894 2024-09-17T16:47:24.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:47:24.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574894 2024-09-17T16:47:24.881 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574894 2024-09-17T16:47:24.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574894' 2024-09-17T16:47:24.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:47:25.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574894 -lt 55834574894 2024-09-17T16:47:25.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:47:25.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345960 2024-09-17T16:47:25.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:47:25.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:47:25.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345960 2024-09-17T16:47:25.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:47:25.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345960 2024-09-17T16:47:25.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345960' 2024-09-17T16:47:25.271 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 85899345960 2024-09-17T16:47:25.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:47:25.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345960 -lt 85899345960 2024-09-17T16:47:25.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:47:25.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018908 2024-09-17T16:47:25.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:47:25.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:47:25.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018908 2024-09-17T16:47:25.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:47:25.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018908 2024-09-17T16:47:25.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018908' 2024-09-17T16:47:25.718 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 128849018908 2024-09-17T16:47:25.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:47:26.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018908 -lt 128849018908 2024-09-17T16:47:26.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:47:26.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-180388626453 2024-09-17T16:47:26.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:47:26.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:47:26.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-180388626453 2024-09-17T16:47:26.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:47:26.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=180388626453 2024-09-17T16:47:26.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 180388626453' 2024-09-17T16:47:26.149 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 180388626453 2024-09-17T16:47:26.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:47:26.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 180388626453 -lt 180388626453 2024-09-17T16:47:26.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:47:26.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-223338299412 2024-09-17T16:47:26.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:47:26.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:47:26.577 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-223338299412 2024-09-17T16:47:26.577 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:47:26.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=223338299412 2024-09-17T16:47:26.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 223338299412' 2024-09-17T16:47:26.578 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 223338299412 2024-09-17T16:47:26.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:47:26.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 223338299413 -lt 223338299412 2024-09-17T16:47:26.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:47:26.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-261993005063 2024-09-17T16:47:26.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:47:26.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T16:47:26.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-261993005063 2024-09-17T16:47:26.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:47:26.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=261993005063 2024-09-17T16:47:26.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 261993005063' 2024-09-17T16:47:26.980 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 261993005063 2024-09-17T16:47:26.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T16:47:27.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 261993005063 -lt 261993005063 2024-09-17T16:47:27.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:47:27.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:47:27.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:47:27.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:47:27.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:47:27.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:47:27.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:47:27.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:47:27.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:47:27.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:47:27.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:47:28.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:47:28.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:47:28.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:47:28.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:47:28.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:47:28.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:47:28.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:47:28.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:418: TEST_ec_single_recovery_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-17T16:47:28.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:47:28.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:47:28.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-17T16:47:28.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:47:28.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:47:28.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:47:28.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:47:28.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:47:28.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:47:28.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:47:28.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:47:28.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-17T16:47:29.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:419: TEST_ec_single_recovery_error: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-17T16:47:29.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T16:47:29.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T16:47:29.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T16:47:29.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T16:47:29.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T16:47:29.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T16:47:29.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-17T16:47:29.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T16:47:29.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T16:47:29.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T16:47:29.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-17T16:47:29.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T16:47:29.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-17T16:47:29.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:47:29.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-17T16:47:29.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-17T16:47:29.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:47:29.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:47:29.444 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:47:29.444 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:47:29.444 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:47:29.445 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-17T16:47:29.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2024-09-17T16:47:29.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-09-17T16:47:29.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T16:47:29.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-17T16:47:29.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T16:47:29.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-17T16:47:29.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T16:47:29.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-17T16:47:29.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T16:47:29.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-17T16:47:29.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T16:47:29.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T16:47:29.447 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T16:47:29.447 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-17T16:47:29.447 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-17T16:47:29.447 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-17T16:47:29.447 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:47:29.447 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:47:29.447 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:47:29.447 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.3.asok 2024-09-17T16:47:29.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-17T16:47:29.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T16:47:29.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T16:47:29.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T16:47:29.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-17T16:47:29.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-17T16:47:29.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-17T16:47:29.537 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:47:29.537 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:47:29.537 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:47:29.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.3.asok 2024-09-17T16:47:29.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T16:47:29.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-17T16:47:29.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_single_recovery_error: get_osds pool-jerasure myobject 2024-09-17T16:47:29.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:47:29.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-17T16:47:29.622 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-17T16:47:29.622 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:47:30.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:47:30.068 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:47:30.068 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:47:30.069 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:47:30.069 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-17T16:47:30.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2024-09-17T16:47:30.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_single_recovery_error: initial_osds=('3' '5' '1' '0' '6') 2024-09-17T16:47:30.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_single_recovery_error: local -a initial_osds 2024-09-17T16:47:30.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:422: TEST_ec_single_recovery_error: local last_osd=6 2024-09-17T16:47:30.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:424: TEST_ec_single_recovery_error: kill_daemons td/test-erasure-eio TERM osd.6 2024-09-17T16:47:30.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:47:30.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:47:30.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:47:30.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:47:30.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:47:30.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:47:30.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:425: TEST_ec_single_recovery_error: ceph osd down 6 2024-09-17T16:47:30.929 INFO:tasks.workunit.client.0.smithi162.stderr:osd.6 is already down. 2024-09-17T16:47:30.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:426: TEST_ec_single_recovery_error: ceph osd out 6 2024-09-17T16:47:31.626 INFO:tasks.workunit.client.0.smithi162.stderr:osd.6 is already out. 2024-09-17T16:47:31.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:429: TEST_ec_single_recovery_error: wait_for_clean 2024-09-17T16:47:31.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:47:31.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:47:31.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:47:31.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:47:31.638 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:47:31.638 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:47:31.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:47:31.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:47:31.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:47:31.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:47:31.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:47:31.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:47:31.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:47:31.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:47:31.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:47:32.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:47:32.166 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:47:32.166 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:47:32.166 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:47:32.167 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:47:32.167 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-09-17T16:47:32.167 INFO:tasks.workunit.client.0.smithi162.stderr:6' 2024-09-17T16:47:32.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:47:32.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:47:32.167 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:47:32.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803835 2024-09-17T16:47:32.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803835 2024-09-17T16:47:32.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835' 2024-09-17T16:47:32.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:47:32.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:47:32.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574898 2024-09-17T16:47:32.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574898 2024-09-17T16:47:32.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574898' 2024-09-17T16:47:32.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:47:32.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:47:32.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345964 2024-09-17T16:47:32.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345964 2024-09-17T16:47:32.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574898 2-85899345964' 2024-09-17T16:47:32.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:47:32.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:47:32.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018912 2024-09-17T16:47:32.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018912 2024-09-17T16:47:32.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574898 2-85899345964 3-128849018912' 2024-09-17T16:47:32.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:47:32.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T16:47:33.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=180388626457 2024-09-17T16:47:33.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 180388626457 2024-09-17T16:47:33.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574898 2-85899345964 3-128849018912 4-180388626457' 2024-09-17T16:47:33.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:47:33.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T16:47:33.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=223338299416 2024-09-17T16:47:33.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 223338299416 2024-09-17T16:47:33.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574898 2-85899345964 3-128849018912 4-180388626457 5-223338299416' 2024-09-17T16:47:33.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:47:33.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T16:47:33.377 INFO:tasks.workunit.client.0.smithi162.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-17T16:47:33.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-17T16:47:33.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-17T16:47:33.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-17T16:47:33.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:47:33.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803835 2024-09-17T16:47:33.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:47:33.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:47:33.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803835 2024-09-17T16:47:33.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:47:33.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803835 2024-09-17T16:47:33.382 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803835 2024-09-17T16:47:33.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803835' 2024-09-17T16:47:33.383 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:47:33.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803833 -lt 25769803835 2024-09-17T16:47:33.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:47:34.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:47:34.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:47:35.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803835 -lt 25769803835 2024-09-17T16:47:35.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:47:35.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574898 2024-09-17T16:47:35.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:47:35.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:47:35.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574898 2024-09-17T16:47:35.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:47:35.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574898 2024-09-17T16:47:35.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574898' 2024-09-17T16:47:35.241 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574898 2024-09-17T16:47:35.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:47:35.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574898 -lt 55834574898 2024-09-17T16:47:35.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:47:35.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345964 2024-09-17T16:47:35.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:47:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:47:35.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345964 2024-09-17T16:47:35.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:47:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345964 2024-09-17T16:47:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345964' 2024-09-17T16:47:35.667 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 85899345964 2024-09-17T16:47:35.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:47:36.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345965 -lt 85899345964 2024-09-17T16:47:36.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:47:36.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018912 2024-09-17T16:47:36.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:47:36.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:47:36.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018912 2024-09-17T16:47:36.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:47:36.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018912 2024-09-17T16:47:36.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018912' 2024-09-17T16:47:36.058 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 128849018912 2024-09-17T16:47:36.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:47:36.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018912 -lt 128849018912 2024-09-17T16:47:36.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:47:36.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-180388626457 2024-09-17T16:47:36.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:47:36.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T16:47:36.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-180388626457 2024-09-17T16:47:36.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:47:36.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=180388626457 2024-09-17T16:47:36.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 180388626457' 2024-09-17T16:47:36.486 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 180388626457 2024-09-17T16:47:36.486 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T16:47:36.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 180388626457 -lt 180388626457 2024-09-17T16:47:36.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:47:36.873 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-223338299416 2024-09-17T16:47:36.873 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:47:36.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T16:47:36.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-223338299416 2024-09-17T16:47:36.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:47:36.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=223338299416 2024-09-17T16:47:36.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 223338299416' 2024-09-17T16:47:36.877 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 223338299416 2024-09-17T16:47:36.877 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T16:47:37.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 223338299417 -lt 223338299416 2024-09-17T16:47:37.302 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:47:37.302 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:47:37.302 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:47:37.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:47:37.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:47:37.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:47:37.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:47:37.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:47:37.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:47:37.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:47:37.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:47:38.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:47:38.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:47:38.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:47:38.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:47:38.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:47:38.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:47:38.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:47:38.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:431: TEST_ec_single_recovery_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-17T16:47:38.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:47:38.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:47:38.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-17T16:47:38.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:47:38.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:47:38.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-17T16:47:38.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:47:38.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:47:38.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_single_recovery_error: delete_erasure_coded_pool pool-jerasure 2024-09-17T16:47:38.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:47:38.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-17T16:47:39.291 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-17T16:47:39.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-17T16:47:39.905 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-17T16:47:39.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-17T16:47:39.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:47:39.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:47:39.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:47:39.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:47:39.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:47:39.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:47:39.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:47:39.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:47:40.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:47:40.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:47:40.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:47:40.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:47:40.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:47:40.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:47:40.032 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:47:40.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:47:40.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:47:40.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:47:40.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:47:40.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:47:40.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:47:40.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:47:40.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:47:40.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:47:40.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:47:40.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:47:40.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:47:40.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:47:40.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-17T16:47:40.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-17T16:47:40.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-17T16:47:40.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-17T16:47:40.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:47:40.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:47:40.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:47:40.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:47:40.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:47:40.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:47:40.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:47:40.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:47:40.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:47:40.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:47:40.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:47:40.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:47:40.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:47:40.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:47:40.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:47:40.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:47:40.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:47:40.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:47:40.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:47:40.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:47:40.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:47:40.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:47:40.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:47:40.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:47:40.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:47:40.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:47:40.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:47:40.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:47:40.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-17T16:47:40.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:47:40.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:47:40.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:47:40.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.87940 2024-09-17T16:47:40.083 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:47:40.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T16:47:40.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:47:40.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-17T16:47:40.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-17T16:47:40.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-17T16:47:40.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:47:40.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:47:40.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:47:40.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-17T16:47:40.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-17T16:47:40.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:47:40.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:47:40.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:47:40.547 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:47:40.548 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:47:40.548 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:47:40.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:47:40.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T16:47:40.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:47:40.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:47:40.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:47:40.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:47:40.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:47:40.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:47:40.774 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:47:40.775 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:47:40.775 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:47:40.775 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:47:40.775 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:47:40.776 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:47:40.776 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:47:40.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:47:40.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get fsid 2024-09-17T16:47:40.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:47:40.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:47:40.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:47:40.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:47:40.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:47:40.856 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:47:40.856 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:47:40.856 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:47:40.856 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:47:40.856 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:47:40.856 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:47:40.857 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:47:40.857 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:47:40.857 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get mon_host 2024-09-17T16:47:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-17T16:47:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-17T16:47:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:47:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:47:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:47:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-17T16:47:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:47:41.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:47:41.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:47:41.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:47:41.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:47:41.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:47:41.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:47:41.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:47:41.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:47:41.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:47:41.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-17T16:47:41.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T16:47:42.097 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T16:47:42.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:47:43.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-17T16:47:43.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:47:43.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:47:43.109 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:47:43.109 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:47:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:47:43.110 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:47:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-17T16:47:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-mon.a.asok log flush 2024-09-17T16:47:43.182 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:47:43.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-17T16:47:43.191 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:47:40.579+0000 7f8ff5021c80 0 load: jerasure load: lrc 2024-09-17T16:47:43.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_0 td/test-erasure-eio 2024-09-17T16:47:43.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:322: TEST_rados_get_bad_size_shard_0: local dir=td/test-erasure-eio 2024-09-17T16:47:43.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_0: setup_osds 4 2024-09-17T16:47:43.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-17T16:47:43.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-17T16:47:43.192 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-17T16:47:43.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-17T16:47:43.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:47:43.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-17T16:47:43.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:47:43.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:47:43.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:47:43.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:47:43.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-17T16:47:43.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:47:43.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:47:43.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:47:43.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:47:43.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-17T16:47:43.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-17T16:47:43.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:47:43.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:47:43.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:47:43.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:47:43.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:47:43.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:47:43.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:47:43.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:47:43.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:47:43.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:47:43.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:47:43.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:47:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:47:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:47:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:47:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:47:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:47:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:47:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:47:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:47:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:47:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:47:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-17T16:47:43.200 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:47:43.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0efca138-8d71-4e03-9bed-7046ad0cddc6 2024-09-17T16:47:43.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0efca138-8d71-4e03-9bed-7046ad0cddc6' 2024-09-17T16:47:43.201 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 0efca138-8d71-4e03-9bed-7046ad0cddc6 2024-09-17T16:47:43.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:47:43.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCvsulmZrHODBAAlLXuwYGL3c4u1ohZ1C8aMw== 2024-09-17T16:47:43.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCvsulmZrHODBAAlLXuwYGL3c4u1ohZ1C8aMw=="}' 2024-09-17T16:47:43.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0efca138-8d71-4e03-9bed-7046ad0cddc6 -i td/test-erasure-eio/0/new.json 2024-09-17T16:47:43.482 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:47:43.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-17T16:47:43.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvsulmZrHODBAAlLXuwYGL3c4u1ohZ1C8aMw== --osd-uuid 0efca138-8d71-4e03-9bed-7046ad0cddc6 2024-09-17T16:47:43.521 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:47:43.518+0000 7f317bbc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:47:43.550 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:47:43.547+0000 7f317bbc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:47:43.551 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:47:43.549+0000 7f317bbc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:47:43.552 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:47:43.549+0000 7f317bbc5640 -1 bdev(0x561573cc1c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:47:43.552 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:47:43.549+0000 7f317bbc5640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-17T16:47:49.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-17T16:47:49.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:47:49.747 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:47:49.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:47:49.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:47:50.280 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:47:50.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:47:50.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:47:50.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:47:50.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:47:50.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:47:50.307 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:47:50.305+0000 7f8532c33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:47:50.362 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:47:50.360+0000 7f8532c33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:47:50.364 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:47:50.362+0000 7f8532c33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:47:50.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:47:50.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:47:50.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:47:50.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:47:50.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:47:50.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:50.673 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:47:50.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:47:50.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:50.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:47:51.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:51.949 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:47:51.947+0000 7f8532c33640 -1 Falling back to public interface 2024-09-17T16:47:52.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:52.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:52.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:47:52.095 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:47:52.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:52.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:47:52.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:53.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:53.539 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:47:53.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:53.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:47:53.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:53.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:47:53.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:54.549 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:47:54.547+0000 7f8532c33640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:47:54.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:54.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:54.935 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:47:54.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:47:54.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:54.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:47:55.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:56.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:56.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:56.469 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:47:56.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:47:56.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:56.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:47:56.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:57.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:58.348 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:47:58.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:58.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:47:58.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:58.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:47:58.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:47:59.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:47:59.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:47:59.396 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:47:59.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:47:59.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:47:59.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:47:59.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:00.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:00.802 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:48:00.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:00.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:48:00.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:00.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:48:01.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:02.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:02.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:02.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:48:02.326 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:48:02.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:02.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:48:02.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:03.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:03.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:03.726 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:48:03.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:48:03.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:03.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:48:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:05.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:05.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:05.303 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:48:05.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:48:05.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:05.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:48:05.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:06.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:06.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:06.724 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:48:06.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:48:06.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:06.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:48:07.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:08.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:08.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:08.407 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:48:08.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:48:08.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:08.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:48:08.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:09.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:09.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:09.932 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:48:09.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:48:09.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:09.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:48:11.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:12.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:12.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:12.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:48:12.127 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:48:12.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:12.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:48:12.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:13.626 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:48:13.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:48:13.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:13.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:48:14.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:15.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:15.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:15.144 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:48:15.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:48:15.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:15.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:48:15.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:16.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:16.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:16.900 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:48:16.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:48:16.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:16.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:48:17.534 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4171316756,v1:127.0.0.1:6803/4171316756] [v2:127.0.0.1:6804/4171316756,v1:127.0.0.1:6805/4171316756] exists,up 0efca138-8d71-4e03-9bed-7046ad0cddc6 2024-09-17T16:48:17.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:48:17.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:48:17.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:48:17.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:48:17.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-17T16:48:17.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:48:17.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:48:17.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T16:48:17.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:48:17.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:48:17.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:48:17.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:48:17.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:48:17.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:48:17.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:48:17.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:48:17.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:48:17.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:48:17.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:48:17.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:48:17.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:48:17.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:48:17.540 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:48:17.540 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:48:17.540 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:48:17.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:48:17.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:48:17.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:48:17.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:48:17.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:48:17.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:48:17.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:48:17.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:48:17.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:48:17.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:48:17.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:48:17.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:48:17.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:48:17.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:48:17.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:48:17.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=792c342f-ff50-45e7-ba85-c4d65b739cb4 2024-09-17T16:48:17.542 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 792c342f-ff50-45e7-ba85-c4d65b739cb4 2024-09-17T16:48:17.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 792c342f-ff50-45e7-ba85-c4d65b739cb4' 2024-09-17T16:48:17.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:48:17.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDRsulmXVwOIRAAIDQGwwaU4VITxFtGOWusEQ== 2024-09-17T16:48:17.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDRsulmXVwOIRAAIDQGwwaU4VITxFtGOWusEQ=="}' 2024-09-17T16:48:17.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 792c342f-ff50-45e7-ba85-c4d65b739cb4 -i td/test-erasure-eio/1/new.json 2024-09-17T16:48:18.345 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:48:18.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-17T16:48:18.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRsulmXVwOIRAAIDQGwwaU4VITxFtGOWusEQ== --osd-uuid 792c342f-ff50-45e7-ba85-c4d65b739cb4 2024-09-17T16:48:18.386 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:48:18.384+0000 7f80df745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:48:18.405 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:48:18.400+0000 7f80df745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:48:18.405 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:48:18.402+0000 7f80df745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:48:18.405 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:48:18.402+0000 7f80df745640 -1 bdev(0x558741211c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-17T16:48:18.405 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:48:18.402+0000 7f80df745640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-17T16:48:25.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-17T16:48:25.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:48:25.245 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T16:48:25.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T16:48:25.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:48:25.786 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T16:48:25.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T16:48:25.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:48:25.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:48:25.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:48:25.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:48:25.814 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:48:25.811+0000 7fcc226f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:48:25.827 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:48:25.825+0000 7fcc226f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:48:25.828 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:48:25.826+0000 7fcc226f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:48:26.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T16:48:26.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:48:26.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:48:26.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:48:26.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:48:26.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:26.252 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:48:26.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:48:26.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:26.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:26.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:26.932 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:48:26.929+0000 7fcc226f1640 -1 Falling back to public interface 2024-09-17T16:48:27.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:27.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:27.696 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:48:27.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:48:27.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:27.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:28.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:29.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:29.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:29.164 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:48:29.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:48:29.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:29.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:29.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:30.317 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:48:30.314+0000 7fcc226f1640 -1 osd.1 0 log_to_monitors true 2024-09-17T16:48:30.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:30.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:30.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:48:30.607 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:48:30.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:30.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:31.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:32.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:32.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:32.139 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:48:32.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:48:32.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:32.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:32.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:33.610 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:48:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:48:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:34.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:35.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:35.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:35.066 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:48:35.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:48:35.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:35.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:35.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:36.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:36.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:36.544 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:48:36.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:48:36.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:36.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:36.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:37.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:37.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:37.979 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:48:37.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:48:37.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:37.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:38.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:39.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:39.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:39.402 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:48:39.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:48:39.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:39.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:39.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:40.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:40.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:40.807 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:48:40.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:48:40.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:40.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:41.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:42.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:42.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:42.327 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:48:42.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:48:42.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:42.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:42.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:43.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:43.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:43.769 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:48:43.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:48:43.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:43.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:44.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:45.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:45.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:45.488 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:48:45.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:48:45.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:45.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:46.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:47.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:47.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:47.191 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:48:47.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:48:47.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:47.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:48.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:49.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:49.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:49.311 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:48:49.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:48:49.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:49.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:50.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:51.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:51.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:51.105 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:48:51.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:48:51.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:51.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:52.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:48:53.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:48:53.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:48:53.435 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:48:53.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:48:53.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:48:53.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:48:54.114 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3724289472,v1:127.0.0.1:6811/3724289472] [v2:127.0.0.1:6812/3724289472,v1:127.0.0.1:6813/3724289472] exists,up 792c342f-ff50-45e7-ba85-c4d65b739cb4 2024-09-17T16:48:54.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:48:54.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:48:54.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:48:54.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:48:54.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-17T16:48:54.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:48:54.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:48:54.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T16:48:54.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:48:54.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:48:54.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:48:54.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:48:54.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:48:54.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:48:54.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:48:54.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:48:54.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:48:54.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:48:54.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:48:54.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:48:54.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:48:54.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:48:54.121 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:48:54.121 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:48:54.121 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:48:54.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:48:54.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:48:54.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:48:54.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:48:54.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:48:54.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:48:54.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:48:54.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:48:54.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:48:54.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:48:54.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:48:54.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:48:54.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:48:54.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:48:54.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:48:54.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=348891fb-3d5a-4a4a-b98f-e7833314e013 2024-09-17T16:48:54.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 348891fb-3d5a-4a4a-b98f-e7833314e013' 2024-09-17T16:48:54.123 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 348891fb-3d5a-4a4a-b98f-e7833314e013 2024-09-17T16:48:54.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:48:54.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD2sulmck8LCBAAYa+1OrbrIlqClvdsxeZ0jw== 2024-09-17T16:48:54.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD2sulmck8LCBAAYa+1OrbrIlqClvdsxeZ0jw=="}' 2024-09-17T16:48:54.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 348891fb-3d5a-4a4a-b98f-e7833314e013 -i td/test-erasure-eio/2/new.json 2024-09-17T16:48:54.755 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:48:54.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-17T16:48:54.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2sulmck8LCBAAYa+1OrbrIlqClvdsxeZ0jw== --osd-uuid 348891fb-3d5a-4a4a-b98f-e7833314e013 2024-09-17T16:48:54.797 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:48:54.795+0000 7fe661c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:48:54.874 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:48:54.872+0000 7fe661c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:48:54.875 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:48:54.873+0000 7fe661c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:48:54.875 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:48:54.873+0000 7fe661c0e640 -1 bdev(0x5565ba3adc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-17T16:48:54.875 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:48:54.873+0000 7fe661c0e640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-17T16:49:01.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-17T16:49:01.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:49:01.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T16:49:01.269 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T16:49:01.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:49:01.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T16:49:01.808 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T16:49:01.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:49:01.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:49:01.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:49:01.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:49:01.836 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:49:01.834+0000 7f5f87f4b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:49:01.874 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:49:01.872+0000 7f5f87f4b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:49:01.874 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:49:01.873+0000 7f5f87f4b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:49:02.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T16:49:02.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:49:02.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:49:02.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:49:02.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:49:02.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:02.205 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:49:02.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:49:02.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:02.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:02.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:03.457 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:49:03.455+0000 7f5f87f4b640 -1 Falling back to public interface 2024-09-17T16:49:03.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:03.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:03.613 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:49:03.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:49:03.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:03.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:04.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:05.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:05.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:05.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:49:05.025 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:49:05.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:05.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:05.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:06.280 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:49:06.278+0000 7f5f87f4b640 -1 osd.2 0 log_to_monitors true 2024-09-17T16:49:06.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:06.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:06.459 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:49:06.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:49:06.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:06.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:06.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:07.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:07.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:07.985 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:49:07.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:49:07.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:07.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:08.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:09.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:09.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:09.480 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:49:09.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:49:09.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:09.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:09.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:10.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:10.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:10.904 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:49:10.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:49:10.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:10.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:11.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:12.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:12.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:12.416 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:49:12.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:49:12.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:12.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:12.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:13.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:13.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:13.814 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:49:13.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:49:13.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:13.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:14.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:15.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:15.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:15.396 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:49:15.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:49:15.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:15.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:15.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:16.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:16.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:16.793 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:49:16.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:49:16.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:16.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:17.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:18.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:18.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:18.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:49:18.242 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:49:18.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:18.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:19.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:20.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:20.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:20.155 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:49:20.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:49:20.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:20.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:20.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:21.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:21.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:21.703 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:49:21.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:49:21.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:21.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:22.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:23.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:23.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:23.566 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:49:23.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:49:23.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:23.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:24.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:25.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:25.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:25.305 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:49:25.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:49:25.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:25.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:26.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:27.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:27.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:27.402 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:49:27.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:49:27.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:27.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:28.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:29.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:29.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:29.154 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:49:29.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:49:29.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:29.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:49:29.607 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3390833525,v1:127.0.0.1:6819/3390833525] [v2:127.0.0.1:6820/3390833525,v1:127.0.0.1:6821/3390833525] exists,up 348891fb-3d5a-4a4a-b98f-e7833314e013 2024-09-17T16:49:29.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:49:29.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:49:29.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:49:29.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:49:29.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-17T16:49:29.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:49:29.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:49:29.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T16:49:29.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:49:29.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T16:49:29.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:49:29.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:49:29.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:49:29.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:49:29.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T16:49:29.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T16:49:29.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:49:29.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:49:29.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:49:29.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:49:29.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:49:29.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:49:29.612 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:49:29.612 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:49:29.613 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:49:29.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:49:29.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:49:29.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:49:29.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:49:29.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:49:29.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:49:29.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:49:29.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:49:29.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:49:29.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:49:29.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:49:29.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:49:29.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:49:29.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T16:49:29.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:49:29.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=52a64aab-eb89-4359-b407-891acba49384 2024-09-17T16:49:29.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 52a64aab-eb89-4359-b407-891acba49384' 2024-09-17T16:49:29.614 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 52a64aab-eb89-4359-b407-891acba49384 2024-09-17T16:49:29.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:49:29.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAZs+lm1E5kJRAAvC4SZvH0h3Gu9JGcXYkBfA== 2024-09-17T16:49:29.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAZs+lm1E5kJRAAvC4SZvH0h3Gu9JGcXYkBfA=="}' 2024-09-17T16:49:29.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 52a64aab-eb89-4359-b407-891acba49384 -i td/test-erasure-eio/3/new.json 2024-09-17T16:49:30.766 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:49:30.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-17T16:49:30.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZs+lm1E5kJRAAvC4SZvH0h3Gu9JGcXYkBfA== --osd-uuid 52a64aab-eb89-4359-b407-891acba49384 2024-09-17T16:49:30.808 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:49:30.806+0000 7f4fc6615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:49:30.809 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:49:30.808+0000 7f4fc6615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:49:30.810 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:49:30.809+0000 7f4fc6615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:49:30.811 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:49:30.809+0000 7f4fc6615640 -1 bdev(0x560c72267c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-17T16:49:30.811 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:49:30.809+0000 7f4fc6615640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-17T16:49:37.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-17T16:49:37.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:49:37.646 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-17T16:49:37.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T16:49:37.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:49:38.197 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T16:49:38.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T16:49:38.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:49:38.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:49:38.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:49:38.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:49:38.224 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:49:38.222+0000 7fcfb1818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:49:38.249 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:49:38.247+0000 7fcfb1818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:49:38.250 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:49:38.248+0000 7fcfb1818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:49:38.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T16:49:38.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:49:38.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:49:38.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:49:38.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:49:38.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:38.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:49:38.622 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:49:38.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:38.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:49:39.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:39.591 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:49:39.589+0000 7fcfb1818640 -1 Falling back to public interface 2024-09-17T16:49:40.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:40.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:40.049 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:49:40.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:49:40.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:40.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:49:40.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:41.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:41.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:41.476 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:49:41.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:49:41.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:41.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:49:41.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:49:42.474+0000 7fcfb1818640 -1 osd.3 0 log_to_monitors true 2024-09-17T16:49:42.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:42.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:42.877 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:49:42.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:49:42.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:42.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:49:43.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:44.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:44.414 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:49:44.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:44.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:49:44.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:44.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:49:45.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:46.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:46.106 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:49:46.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:46.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:49:46.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:46.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:49:46.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:47.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:47.659 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:49:47.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:47.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:49:47.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:47.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:49:48.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:49.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:49.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:49.069 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:49:49.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:49:49.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:49.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:49:49.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:50.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:50.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:50.486 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:49:50.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:49:50.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:50.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:49:50.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:51.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:51.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:51.926 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:49:51.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:49:51.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:51.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:49:52.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:53.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:53.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:53.397 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:49:53.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:49:53.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:53.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:49:53.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:54.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:54.821 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:49:54.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:54.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:49:54.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:54.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:49:55.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:56.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:56.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:56.326 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:49:56.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:49:56.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:56.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:49:57.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:49:58.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:49:58.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:49:58.111 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:49:58.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:49:58.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:49:58.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:49:59.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:50:00.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:50:00.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:00.066 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:50:00.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:50:00.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:00.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:50:00.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:50:01.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:50:01.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:01.880 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:50:01.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:50:01.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:01.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:50:02.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:50:03.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:50:03.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:03.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:50:03.512 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:50:03.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:03.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:50:04.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:50:05.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:50:05.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:05.308 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:50:05.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:50:05.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:05.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:50:06.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:50:07.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:50:07.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:07.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:50:07.228 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:50:07.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:07.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:50:07.624 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2176185096,v1:127.0.0.1:6827/2176185096] [v2:127.0.0.1:6828/2176185096,v1:127.0.0.1:6829/2176185096] exists,up 52a64aab-eb89-4359-b407-891acba49384 2024-09-17T16:50:07.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:50:07.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:50:07.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:50:07.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-17T16:50:07.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:50:07.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:50:07.626 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:50:07.626 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:50:07.626 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:50:07.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T16:50:07.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-17T16:50:07.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.0.asok log flush 2024-09-17T16:50:07.706 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:50:07.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-17T16:50:07.716 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:47:51.952+0000 7f8532c33640 0 load: jerasure load: lrc 2024-09-17T16:50:07.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:325: TEST_rados_get_bad_size_shard_0: local poolname=pool-jerasure 2024-09-17T16:50:07.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:326: TEST_rados_get_bad_size_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-17T16:50:07.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:50:07.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-17T16:50:07.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-17T16:50:07.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-17T16:50:07.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-17T16:50:07.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-17T16:50:07.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-17T16:50:08.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-17T16:50:08.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-17T16:50:09.560 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-17T16:50:09.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:50:10.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-17T16:50:10.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:50:10.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:50:10.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:50:10.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:50:10.574 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:50:10.574 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:50:10.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:50:10.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:50:10.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:50:10.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:50:10.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:50:10.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:50:10.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:50:10.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:50:10.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:50:11.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:50:11.063 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:50:11.063 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:50:11.063 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:50:11.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:50:11.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:11.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:50:11.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803806 2024-09-17T16:50:11.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803806 2024-09-17T16:50:11.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806' 2024-09-17T16:50:11.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:11.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:50:11.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574868 2024-09-17T16:50:11.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574868 2024-09-17T16:50:11.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806 1-55834574868' 2024-09-17T16:50:11.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:11.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:50:11.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378641 2024-09-17T16:50:11.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378641 2024-09-17T16:50:11.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806 1-55834574868 2-81604378641' 2024-09-17T16:50:11.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:11.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:50:11.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084293 2024-09-17T16:50:11.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084293 2024-09-17T16:50:11.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806 1-55834574868 2-81604378641 3-120259084293' 2024-09-17T16:50:11.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:50:11.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803806 2024-09-17T16:50:11.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:50:11.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:50:11.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803806 2024-09-17T16:50:11.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:50:11.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803806 2024-09-17T16:50:11.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803806' 2024-09-17T16:50:11.521 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803806 2024-09-17T16:50:11.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:50:11.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803804 -lt 25769803806 2024-09-17T16:50:11.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:50:12.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:50:12.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:50:13.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803806 -lt 25769803806 2024-09-17T16:50:13.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:50:13.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574868 2024-09-17T16:50:13.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:50:13.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:50:13.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574868 2024-09-17T16:50:13.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:50:13.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574868 2024-09-17T16:50:13.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574868' 2024-09-17T16:50:13.332 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574868 2024-09-17T16:50:13.332 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:50:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574868 -lt 55834574868 2024-09-17T16:50:13.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:50:13.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378641 2024-09-17T16:50:13.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:50:13.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:50:13.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378641 2024-09-17T16:50:13.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:50:13.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378641 2024-09-17T16:50:13.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378641' 2024-09-17T16:50:13.760 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378641 2024-09-17T16:50:13.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:50:14.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378641 -lt 81604378641 2024-09-17T16:50:14.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:50:14.167 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084293 2024-09-17T16:50:14.167 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:50:14.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:50:14.169 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084293 2024-09-17T16:50:14.169 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:50:14.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084293 2024-09-17T16:50:14.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084293' 2024-09-17T16:50:14.170 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084293 2024-09-17T16:50:14.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:50:14.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084294 -lt 120259084293 2024-09-17T16:50:14.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:50:14.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:50:14.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:50:15.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:50:15.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:50:15.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:50:15.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:50:15.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:50:15.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:50:15.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:50:15.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:50:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:50:15.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:50:15.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:50:15.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:50:16.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:50:16.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:50:16.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:50:16.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:328: TEST_rados_get_bad_size_shard_0: local shard_id=0 2024-09-17T16:50:16.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:329: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 10 2024-09-17T16:50:16.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-17T16:50:16.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-17T16:50:16.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=0 2024-09-17T16:50:16.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-17T16:50:16.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=10 2024-09-17T16:50:16.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-17T16:50:16.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2024-09-17T16:50:16.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-17T16:50:16.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-87940-0-10 2024-09-17T16:50:16.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-87940-0-10 2024-09-17T16:50:16.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:50:16.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:50:16.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-87940-0-10 2024-09-17T16:50:16.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:50:16.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:50:16.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:50:16.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:50:16.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:50:16.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:50:16.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:50:16.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:50:16.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-87940-0-10 td/test-erasure-eio/ORIGINAL 2024-09-17T16:50:16.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-87940-0-10 td/test-erasure-eio 0 10 set 2024-09-17T16:50:16.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-87940-0-10 2024-09-17T16:50:16.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-17T16:50:16.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-17T16:50:16.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-17T16:50:16.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=0 2024-09-17T16:50:16.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-17T16:50:16.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2024-09-17T16:50:16.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-17T16:50:16.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-17T16:50:16.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-17T16:50:16.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-87940-0-10 2024-09-17T16:50:16.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:50:16.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-87940-0-10 2024-09-17T16:50:16.229 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-87940-0-10 2024-09-17T16:50:16.230 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:50:16.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:50:16.615 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:50:16.616 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:50:16.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T16:50:16.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-17T16:50:16.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-17T16:50:16.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=3 2024-09-17T16:50:16.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-17T16:50:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-17T16:50:17.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-17T16:50:17.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2024-09-17T16:50:17.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-17T16:50:17.226 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-17T16:50:17.226 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-17T16:50:17.226 INFO:tasks.workunit.client.0.smithi162.stderr:10 bytes copied, 4.4155e-05 s, 226 kB/s 2024-09-17T16:50:17.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-87940-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:50:17.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:50:17.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:50:17.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T16:50:17.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:50:17.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-87940-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:50:17.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:50:17.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:50:17.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T16:50:17.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:50:17.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-17T16:50:17.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:50:17.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:50:17.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:50:17.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:50:17.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:50:17.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:50:17.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-87940-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:50:17.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:50:17.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:50:17.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T16:50:17.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:50:17.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-17T16:50:17.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-87940-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:50:19.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-17T16:50:19.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:50:19.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:50:19.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T16:50:19.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:50:19.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T16:50:19.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:50:19.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:50:19.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:50:19.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:50:19.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T16:50:19.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T16:50:19.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:50:19.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:50:19.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:50:19.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:50:19.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:50:19.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:50:19.711 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:50:19.711 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:50:19.711 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:50:19.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:50:19.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:50:19.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:50:19.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:50:19.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:50:19.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:50:19.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:50:19.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:50:19.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:50:19.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:50:19.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:50:19.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T16:50:19.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T16:50:19.713 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-09-17T16:50:19.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:50:19.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-17T16:50:19.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T16:50:19.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:50:19.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:50:19.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:50:19.741 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:50:19.739+0000 7f1154022640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:50:19.743 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:50:19.741+0000 7f1154022640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:50:19.745 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:50:19.743+0000 7f1154022640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:50:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T16:50:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:50:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:50:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:50:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:50:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:50:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:50:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:50:20.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:50:21.360 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:50:21.358+0000 7f1154022640 -1 Falling back to public interface 2024-09-17T16:50:21.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:50:21.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:21.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:50:21.584 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:50:21.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:21.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:50:21.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:50:22.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:50:22.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:22.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:50:22.977 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:50:22.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:22.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:50:23.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:50:24.134 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:50:24.132+0000 7f1154022640 -1 osd.3 35 log_to_monitors true 2024-09-17T16:50:24.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:50:24.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:24.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:50:24.404 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:50:24.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:24.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:50:24.842 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 41 up_thru 33 down_at 38 last_clean_interval [28,35) [v2:127.0.0.1:6826/1893047141,v1:127.0.0.1:6827/1893047141] [v2:127.0.0.1:6828/1893047141,v1:127.0.0.1:6829/1893047141] exists,up 52a64aab-eb89-4359-b407-891acba49384 2024-09-17T16:50:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:50:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:50:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:50:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:50:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:50:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:50:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:50:24.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:50:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:50:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:50:24.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:50:24.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:50:24.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:50:24.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:50:24.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:50:24.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:50:24.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:50:24.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:50:24.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:50:25.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:50:25.590 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:50:25.590 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:50:25.590 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:50:25.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:50:25.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:25.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:50:25.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803811 2024-09-17T16:50:25.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803811 2024-09-17T16:50:25.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811' 2024-09-17T16:50:25.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:25.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:50:25.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574873 2024-09-17T16:50:25.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574873 2024-09-17T16:50:25.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574873' 2024-09-17T16:50:25.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:25.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:50:26.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378646 2024-09-17T16:50:26.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378646 2024-09-17T16:50:26.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574873 2-81604378646' 2024-09-17T16:50:26.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:26.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:50:26.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659139 2024-09-17T16:50:26.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659139 2024-09-17T16:50:26.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574873 2-81604378646 3-176093659139' 2024-09-17T16:50:26.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:50:26.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803811 2024-09-17T16:50:26.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:50:26.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:50:26.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803811 2024-09-17T16:50:26.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:50:26.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803811 2024-09-17T16:50:26.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803811' 2024-09-17T16:50:26.224 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803811 2024-09-17T16:50:26.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:50:26.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803809 -lt 25769803811 2024-09-17T16:50:26.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:50:27.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:50:27.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:50:28.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803811 -lt 25769803811 2024-09-17T16:50:28.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:50:28.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574873 2024-09-17T16:50:28.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:50:28.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:50:28.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574873 2024-09-17T16:50:28.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:50:28.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574873 2024-09-17T16:50:28.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574873' 2024-09-17T16:50:28.041 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 55834574873 2024-09-17T16:50:28.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:50:28.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574873 -lt 55834574873 2024-09-17T16:50:28.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:50:28.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378646 2024-09-17T16:50:28.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:50:28.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:50:28.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378646 2024-09-17T16:50:28.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:50:28.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378646 2024-09-17T16:50:28.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378646' 2024-09-17T16:50:28.460 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 81604378646 2024-09-17T16:50:28.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:50:28.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378646 -lt 81604378646 2024-09-17T16:50:28.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:50:28.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-176093659139 2024-09-17T16:50:28.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:50:28.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:50:28.877 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-176093659139 2024-09-17T16:50:28.877 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:50:28.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659139 2024-09-17T16:50:28.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 176093659139' 2024-09-17T16:50:28.878 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 176093659139 2024-09-17T16:50:28.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:50:29.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659139 -lt 176093659139 2024-09-17T16:50:29.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:50:29.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:50:29.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:50:29.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:50:29.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:50:29.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:50:29.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:50:29.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:50:29.842 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:50:29.842 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:50:29.842 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:50:30.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:50:30.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:50:30.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:50:30.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:50:30.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:50:30.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:50:30.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:50:30.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-17T16:50:30.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-17T16:50:31.545 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-17T16:50:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87940-0-10 2024-09-17T16:50:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:50:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:50:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87940-0-10 2024-09-17T16:50:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:50:31.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:50:31.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-87940-0-10 td/test-erasure-eio/COPY 2024-09-17T16:50:31.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:50:31.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:50:31.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 0 + 1 2024-09-17T16:50:31.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=1 2024-09-17T16:50:31.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-87940-0-10 td/test-erasure-eio 1 10 set 2024-09-17T16:50:31.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-87940-0-10 2024-09-17T16:50:31.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-17T16:50:31.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-17T16:50:31.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-17T16:50:31.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-17T16:50:31.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-17T16:50:31.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2024-09-17T16:50:31.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-17T16:50:31.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-17T16:50:31.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-17T16:50:31.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-87940-0-10 2024-09-17T16:50:31.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:50:31.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-87940-0-10 2024-09-17T16:50:31.755 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-87940-0-10 2024-09-17T16:50:31.755 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:50:32.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:50:32.195 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:50:32.195 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:50:32.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T16:50:32.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-17T16:50:32.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-17T16:50:32.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-17T16:50:32.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-17T16:50:32.792 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-17T16:50:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-17T16:50:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2024-09-17T16:50:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-17T16:50:32.806 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-17T16:50:32.806 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-17T16:50:32.806 INFO:tasks.workunit.client.0.smithi162.stderr:10 bytes copied, 4.3516e-05 s, 230 kB/s 2024-09-17T16:50:32.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87940-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:50:32.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:50:32.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:50:32.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T16:50:32.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:50:32.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87940-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:50:32.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:50:32.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:50:32.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T16:50:32.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:50:32.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-17T16:50:32.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:50:32.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:50:32.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:50:32.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:50:32.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:50:34.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:50:34.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87940-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:50:34.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:50:34.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:50:34.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T16:50:34.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:50:34.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-17T16:50:34.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87940-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:50:36.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-17T16:50:36.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:50:36.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:50:36.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T16:50:36.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:50:36.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:50:36.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:50:36.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:50:36.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:50:36.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:50:36.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:50:36.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:50:36.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:50:36.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:50:36.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:50:36.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:50:36.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:50:36.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:50:36.038 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:50:36.038 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:50:36.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:50:36.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:50:36.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:50:36.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:50:36.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:50:36.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:50:36.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:50:36.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:50:36.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:50:36.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:50:36.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:50:36.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:50:36.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:50:36.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T16:50:36.040 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-17T16:50:36.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:50:36.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-17T16:50:36.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T16:50:36.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:50:36.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:50:36.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:50:36.063 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:50:36.061+0000 7f5186820640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:50:36.093 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:50:36.091+0000 7f5186820640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:50:36.095 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:50:36.092+0000 7f5186820640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:50:36.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T16:50:36.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:50:36.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:50:36.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:50:36.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:50:36.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:36.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:50:36.464 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:50:36.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:36.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:50:36.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:50:37.447 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:50:37.445+0000 7f5186820640 -1 Falling back to public interface 2024-09-17T16:50:37.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:50:37.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:37.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:50:37.890 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:50:37.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:37.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:50:38.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:50:39.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:50:39.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:39.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:50:39.326 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:50:39.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:39.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:50:39.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:50:40.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:50:40.560+0000 7f5186820640 -1 osd.1 47 log_to_monitors true 2024-09-17T16:50:40.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:50:40.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:40.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:50:40.760 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:50:40.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:40.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:50:41.183 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 51 up_thru 38 down_at 48 last_clean_interval [13,47) [v2:127.0.0.1:6810/355632755,v1:127.0.0.1:6811/355632755] [v2:127.0.0.1:6812/355632755,v1:127.0.0.1:6813/355632755] exists,up 792c342f-ff50-45e7-ba85-c4d65b739cb4 2024-09-17T16:50:41.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:50:41.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:50:41.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:50:41.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:50:41.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:50:41.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:50:41.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:50:41.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:50:41.184 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:50:41.184 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:50:41.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:50:41.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:50:41.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:50:41.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:50:41.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:50:41.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:50:41.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:50:41.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:50:41.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:50:41.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:50:41.665 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:50:41.665 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:50:41.665 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:50:41.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:50:41.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:41.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:50:41.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803816 2024-09-17T16:50:41.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803816 2024-09-17T16:50:41.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803816' 2024-09-17T16:50:41.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:41.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:50:42.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332099 2024-09-17T16:50:42.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332099 2024-09-17T16:50:42.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803816 1-219043332099' 2024-09-17T16:50:42.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:42.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:50:42.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378651 2024-09-17T16:50:42.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378651 2024-09-17T16:50:42.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803816 1-219043332099 2-81604378651' 2024-09-17T16:50:42.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:42.401 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:50:42.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659144 2024-09-17T16:50:42.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659144 2024-09-17T16:50:42.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803816 1-219043332099 2-81604378651 3-176093659144' 2024-09-17T16:50:42.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:50:42.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803816 2024-09-17T16:50:42.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:50:42.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:50:42.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803816 2024-09-17T16:50:42.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:50:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803816 2024-09-17T16:50:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803816' 2024-09-17T16:50:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803816 2024-09-17T16:50:42.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:50:43.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803814 -lt 25769803816 2024-09-17T16:50:43.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:50:44.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:50:44.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:50:44.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803817 -lt 25769803816 2024-09-17T16:50:44.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:50:44.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-219043332099 2024-09-17T16:50:44.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:50:44.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:50:44.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-219043332099 2024-09-17T16:50:44.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:50:44.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332099 2024-09-17T16:50:44.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 219043332099' 2024-09-17T16:50:44.483 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 219043332099 2024-09-17T16:50:44.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:50:44.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332099 -lt 219043332099 2024-09-17T16:50:44.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:50:44.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378651 2024-09-17T16:50:44.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:50:44.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:50:44.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378651 2024-09-17T16:50:44.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:50:44.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378651 2024-09-17T16:50:44.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378651' 2024-09-17T16:50:44.908 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 81604378651 2024-09-17T16:50:44.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:50:45.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378651 -lt 81604378651 2024-09-17T16:50:45.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:50:45.332 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-176093659144 2024-09-17T16:50:45.332 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:50:45.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:50:45.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-176093659144 2024-09-17T16:50:45.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:50:45.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659144 2024-09-17T16:50:45.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 176093659144' 2024-09-17T16:50:45.335 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 176093659144 2024-09-17T16:50:45.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:50:45.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659144 -lt 176093659144 2024-09-17T16:50:45.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:50:45.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:50:45.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:50:46.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:50:46.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:50:46.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:50:46.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:50:46.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:50:46.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:50:46.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:50:46.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:50:46.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:50:46.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:50:46.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:50:46.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:50:47.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:50:47.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:50:47.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:50:47.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-17T16:50:47.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-17T16:50:47.925 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-17T16:50:47.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 0 -eq 1 -a 1 -eq 2 ']' 2024-09-17T16:50:47.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87940-0-10 fail 2024-09-17T16:50:47.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:50:47.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:50:47.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87940-0-10 2024-09-17T16:50:47.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-17T16:50:47.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-17T16:50:47.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-87940-0-10 td/test-erasure-eio/COPY 2024-09-17T16:50:48.085 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj-size-87940-0-10: (5) Input/output error 2024-09-17T16:50:48.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-17T16:50:48.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-17T16:50:48.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:330: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 0 2024-09-17T16:50:48.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-17T16:50:48.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-17T16:50:48.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=0 2024-09-17T16:50:48.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-17T16:50:48.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=0 2024-09-17T16:50:48.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-17T16:50:48.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2024-09-17T16:50:48.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-17T16:50:48.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-87940-0-0 2024-09-17T16:50:48.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-87940-0-0 2024-09-17T16:50:48.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:50:48.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:50:48.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-87940-0-0 2024-09-17T16:50:48.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:50:48.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:50:48.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:50:48.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:50:48.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:50:48.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:50:48.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:50:48.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:50:48.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-87940-0-0 td/test-erasure-eio/ORIGINAL 2024-09-17T16:50:48.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-87940-0-0 td/test-erasure-eio 0 0 set 2024-09-17T16:50:48.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-87940-0-0 2024-09-17T16:50:48.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-17T16:50:48.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-17T16:50:48.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-17T16:50:48.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=0 2024-09-17T16:50:48.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-17T16:50:48.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2024-09-17T16:50:48.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-17T16:50:48.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-17T16:50:48.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-17T16:50:48.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-87940-0-0 2024-09-17T16:50:48.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:50:48.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-87940-0-0 2024-09-17T16:50:48.318 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-87940-0-0 2024-09-17T16:50:48.319 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:50:48.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:50:48.717 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:50:48.718 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:50:48.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T16:50:48.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-17T16:50:48.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-17T16:50:48.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=3 2024-09-17T16:50:48.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-17T16:50:49.315 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-17T16:50:49.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-17T16:50:49.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2024-09-17T16:50:49.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-17T16:50:49.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-87940-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:50:49.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:50:49.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:50:49.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T16:50:49.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:50:49.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-87940-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:50:49.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:50:49.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:50:49.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T16:50:49.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:50:49.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-17T16:50:49.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:50:49.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:50:49.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:50:49.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:50:49.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:50:49.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:50:49.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-87940-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:50:49.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:50:49.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:50:49.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T16:50:49.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:50:49.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-17T16:50:49.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-87940-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:50:51.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-17T16:50:51.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:50:51.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:50:51.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T16:50:51.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:50:51.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T16:50:51.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:50:51.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:50:51.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:50:51.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:50:51.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T16:50:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T16:50:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:50:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:50:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:50:51.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:50:51.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:50:51.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:50:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:50:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:50:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:50:51.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:50:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:50:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:50:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:50:51.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:50:51.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:50:51.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:50:51.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:50:51.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:50:51.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:50:51.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:50:51.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T16:50:51.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T16:50:51.673 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-09-17T16:50:51.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:50:51.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-17T16:50:51.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T16:50:51.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:50:51.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:50:51.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:50:51.701 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:50:51.698+0000 7fd39fc24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:50:51.704 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:50:51.701+0000 7fd39fc24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:50:51.705 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:50:51.702+0000 7fd39fc24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:50:52.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T16:50:52.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:50:52.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:50:52.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:50:52.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:50:52.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:52.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:50:52.083 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:50:52.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:52.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:50:52.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:50:52.573 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:50:52.570+0000 7fd39fc24640 -1 Falling back to public interface 2024-09-17T16:50:53.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:50:53.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:53.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:50:53.536 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:50:53.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:53.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:50:53.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:50:54.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:50:54.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:54.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:50:54.994 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:50:54.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:54.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:50:55.100 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:50:55.097+0000 7fd39fc24640 -1 osd.3 58 log_to_monitors true 2024-09-17T16:50:55.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:50:56.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:50:56.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:50:56.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:50:56.427 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:50:56.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:50:56.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:50:56.925 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 62 up_thru 62 down_at 59 last_clean_interval [41,58) [v2:127.0.0.1:6826/2791160137,v1:127.0.0.1:6827/2791160137] [v2:127.0.0.1:6828/2791160137,v1:127.0.0.1:6829/2791160137] exists,up 52a64aab-eb89-4359-b407-891acba49384 2024-09-17T16:50:56.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:50:56.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:50:56.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:50:56.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:50:56.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:50:56.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:50:56.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:50:56.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:50:56.926 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:50:56.926 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:50:56.926 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:50:56.926 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:50:56.926 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:50:57.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:50:57.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:50:57.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:50:57.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:50:57.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:50:57.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:50:57.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:50:57.552 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:50:57.552 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:50:57.552 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:50:57.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:50:57.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:57.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:50:57.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803821 2024-09-17T16:50:57.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803821 2024-09-17T16:50:57.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803821' 2024-09-17T16:50:57.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:57.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:50:57.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332104 2024-09-17T16:50:57.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332104 2024-09-17T16:50:57.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803821 1-219043332104' 2024-09-17T16:50:57.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:57.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:50:58.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378656 2024-09-17T16:50:58.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378656 2024-09-17T16:50:58.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803821 1-219043332104 2-81604378656' 2024-09-17T16:50:58.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:50:58.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:50:58.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=266287972355 2024-09-17T16:50:58.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 266287972355 2024-09-17T16:50:58.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803821 1-219043332104 2-81604378656 3-266287972355' 2024-09-17T16:50:58.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:50:58.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803821 2024-09-17T16:50:58.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:50:58.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:50:58.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803821 2024-09-17T16:50:58.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:50:58.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803821 2024-09-17T16:50:58.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803821' 2024-09-17T16:50:58.120 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803821 2024-09-17T16:50:58.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:50:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803821 -lt 25769803821 2024-09-17T16:50:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:50:58.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-219043332104 2024-09-17T16:50:58.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:50:58.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:50:58.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-219043332104 2024-09-17T16:50:58.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:50:58.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332104 2024-09-17T16:50:58.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 219043332104' 2024-09-17T16:50:58.544 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 219043332104 2024-09-17T16:50:58.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:50:58.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332102 -lt 219043332104 2024-09-17T16:50:58.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:50:59.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:50:59.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:51:00.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332104 -lt 219043332104 2024-09-17T16:51:00.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:00.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378656 2024-09-17T16:51:00.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:00.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:51:00.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378656 2024-09-17T16:51:00.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:00.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378656 2024-09-17T16:51:00.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378656' 2024-09-17T16:51:00.365 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 81604378656 2024-09-17T16:51:00.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:51:00.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378657 -lt 81604378656 2024-09-17T16:51:00.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:00.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-266287972355 2024-09-17T16:51:00.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:00.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:51:00.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-266287972355 2024-09-17T16:51:00.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:00.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=266287972355 2024-09-17T16:51:00.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 266287972355' 2024-09-17T16:51:00.789 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 266287972355 2024-09-17T16:51:00.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:51:01.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 266287972355 -lt 266287972355 2024-09-17T16:51:01.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:51:01.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:51:01.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:51:01.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:51:01.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:51:01.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:51:01.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:51:01.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:51:01.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:51:01.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:51:01.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:51:02.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:51:02.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:51:02.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:51:02.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:51:02.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:51:02.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:51:02.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:51:02.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-17T16:51:02.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-17T16:51:03.287 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-17T16:51:03.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87940-0-0 2024-09-17T16:51:03.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:51:03.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:51:03.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87940-0-0 2024-09-17T16:51:03.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:51:03.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:51:03.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-87940-0-0 td/test-erasure-eio/COPY 2024-09-17T16:51:03.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:51:03.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:51:03.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 0 + 1 2024-09-17T16:51:03.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=1 2024-09-17T16:51:03.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-87940-0-0 td/test-erasure-eio 1 0 set 2024-09-17T16:51:03.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-87940-0-0 2024-09-17T16:51:03.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-17T16:51:03.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-17T16:51:03.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-17T16:51:03.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-17T16:51:03.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-17T16:51:03.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2024-09-17T16:51:03.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-17T16:51:03.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-17T16:51:03.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-17T16:51:03.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-87940-0-0 2024-09-17T16:51:03.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:51:03.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-87940-0-0 2024-09-17T16:51:03.526 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-87940-0-0 2024-09-17T16:51:03.526 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:51:04.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:51:04.012 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:51:04.012 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:51:04.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T16:51:04.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-17T16:51:04.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-17T16:51:04.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-17T16:51:04.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-17T16:51:04.669 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-17T16:51:04.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-17T16:51:04.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2024-09-17T16:51:04.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-17T16:51:04.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87940-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:04.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:51:04.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:51:04.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T16:51:04.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:51:04.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87940-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:04.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:51:04.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:51:04.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T16:51:04.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:51:04.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-17T16:51:04.685 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:51:04.685 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:51:04.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:51:04.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:51:04.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:51:04.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:51:04.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87940-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:04.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:51:04.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:51:04.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T16:51:04.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:51:04.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-17T16:51:04.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87940-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:07.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-17T16:51:07.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:51:07.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:51:07.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T16:51:07.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:51:07.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:51:07.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:51:07.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:51:07.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:51:07.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:51:07.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:51:07.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:51:07.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:51:07.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:51:07.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:51:07.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:51:07.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:51:07.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:51:07.460 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:51:07.460 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:51:07.460 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:51:07.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:51:07.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:51:07.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:51:07.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:51:07.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:51:07.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:51:07.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:51:07.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:51:07.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:51:07.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:51:07.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:51:07.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:51:07.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T16:51:07.462 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-17T16:51:07.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:51:07.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-17T16:51:07.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T16:51:07.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:51:07.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:51:07.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:51:07.490 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:07.488+0000 7f725aed8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:51:07.493 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:07.491+0000 7f725aed8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:51:07.494 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:07.492+0000 7f725aed8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:51:07.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T16:51:07.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:51:07.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:51:07.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:51:07.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:51:07.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:07.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:51:07.856 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:51:07.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:07.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:51:08.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:51:08.575 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:08.573+0000 7f725aed8640 -1 Falling back to public interface 2024-09-17T16:51:09.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:51:09.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:09.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:51:09.282 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:51:09.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:09.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:51:09.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:51:10.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:51:10.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:10.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:51:10.676 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:51:10.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:10.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:51:11.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:51:11.391 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:11.389+0000 7f725aed8640 -1 osd.1 67 log_to_monitors true 2024-09-17T16:51:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:51:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:51:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:51:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:51:12.792 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [51,67) [v2:127.0.0.1:6810/3721086608,v1:127.0.0.1:6811/3721086608] [v2:127.0.0.1:6812/3721086608,v1:127.0.0.1:6813/3721086608] exists,up 792c342f-ff50-45e7-ba85-c4d65b739cb4 2024-09-17T16:51:12.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:51:12.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:51:12.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:51:12.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:51:12.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:51:12.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:51:12.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:51:12.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:51:12.793 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:51:12.793 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:51:12.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:51:12.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:51:12.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:51:12.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:51:12.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:51:12.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:51:12.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:51:12.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:51:12.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:51:13.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:51:13.319 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:51:13.319 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:51:13.319 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:51:13.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:51:13.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:13.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:51:13.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803827 2024-09-17T16:51:13.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803827 2024-09-17T16:51:13.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827' 2024-09-17T16:51:13.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:13.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:51:13.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645315 2024-09-17T16:51:13.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645315 2024-09-17T16:51:13.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-309237645315' 2024-09-17T16:51:13.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:13.669 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:51:13.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378661 2024-09-17T16:51:13.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378661 2024-09-17T16:51:13.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-309237645315 2-81604378661' 2024-09-17T16:51:13.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:13.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:51:13.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=266287972360 2024-09-17T16:51:13.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 266287972360 2024-09-17T16:51:13.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-309237645315 2-81604378661 3-266287972360' 2024-09-17T16:51:13.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:13.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803827 2024-09-17T16:51:13.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:13.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:51:13.895 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803827 2024-09-17T16:51:13.895 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:13.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803827 2024-09-17T16:51:13.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803827' 2024-09-17T16:51:13.896 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803827 2024-09-17T16:51:13.896 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:51:14.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803827 -lt 25769803827 2024-09-17T16:51:14.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:14.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-309237645315 2024-09-17T16:51:14.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:14.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:51:14.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-309237645315 2024-09-17T16:51:14.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:14.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645315 2024-09-17T16:51:14.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 309237645315' 2024-09-17T16:51:14.349 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 309237645315 2024-09-17T16:51:14.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:51:14.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645315 -lt 309237645315 2024-09-17T16:51:14.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:14.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378661 2024-09-17T16:51:14.815 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:14.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:51:14.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378661 2024-09-17T16:51:14.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:14.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378661 2024-09-17T16:51:14.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378661' 2024-09-17T16:51:14.818 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 81604378661 2024-09-17T16:51:14.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:51:15.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378661 -lt 81604378661 2024-09-17T16:51:15.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:15.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-266287972360 2024-09-17T16:51:15.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:15.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:51:15.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-266287972360 2024-09-17T16:51:15.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:15.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=266287972360 2024-09-17T16:51:15.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 266287972360' 2024-09-17T16:51:15.243 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 266287972360 2024-09-17T16:51:15.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:51:15.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 266287972358 -lt 266287972360 2024-09-17T16:51:15.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:51:16.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:51:16.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:51:17.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 266287972361 -lt 266287972360 2024-09-17T16:51:17.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:51:17.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:51:17.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:51:17.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:51:17.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:51:17.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:51:17.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:51:17.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:51:17.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:51:17.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:51:17.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:51:18.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:51:18.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:51:18.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:51:18.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:51:18.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:51:18.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:51:18.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:51:18.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-17T16:51:18.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-17T16:51:19.205 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-17T16:51:19.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 0 -eq 1 -a 1 -eq 2 ']' 2024-09-17T16:51:19.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87940-0-0 fail 2024-09-17T16:51:19.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:51:19.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:51:19.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87940-0-0 2024-09-17T16:51:19.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-17T16:51:19.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-17T16:51:19.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-87940-0-0 td/test-erasure-eio/COPY 2024-09-17T16:51:19.382 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj-size-87940-0-0: (5) Input/output error 2024-09-17T16:51:19.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-17T16:51:19.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-17T16:51:19.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:331: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 256 add 2024-09-17T16:51:19.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-17T16:51:19.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-17T16:51:19.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=0 2024-09-17T16:51:19.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-17T16:51:19.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=256 2024-09-17T16:51:19.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-17T16:51:19.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=add 2024-09-17T16:51:19.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-17T16:51:19.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-87940-0-256 2024-09-17T16:51:19.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-87940-0-256 2024-09-17T16:51:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:51:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:51:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-87940-0-256 2024-09-17T16:51:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:51:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:51:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:51:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:51:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:51:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:51:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:51:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:51:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-87940-0-256 td/test-erasure-eio/ORIGINAL 2024-09-17T16:51:19.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-87940-0-256 td/test-erasure-eio 0 256 add 2024-09-17T16:51:19.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-87940-0-256 2024-09-17T16:51:19.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-17T16:51:19.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-17T16:51:19.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-17T16:51:19.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=0 2024-09-17T16:51:19.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-17T16:51:19.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2024-09-17T16:51:19.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-17T16:51:19.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2024-09-17T16:51:19.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-17T16:51:19.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-87940-0-256 2024-09-17T16:51:19.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:51:19.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-87940-0-256 2024-09-17T16:51:19.629 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-87940-0-256 2024-09-17T16:51:19.629 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:51:20.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:51:20.040 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:51:20.040 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:51:20.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T16:51:20.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-17T16:51:20.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-17T16:51:20.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=3 2024-09-17T16:51:20.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-17T16:51:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-17T16:51:20.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2024-09-17T16:51:20.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-87940-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:20.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:51:20.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:51:20.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T16:51:20.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:51:20.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-87940-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:20.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:51:20.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:51:20.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T16:51:20.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:51:20.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-17T16:51:20.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:51:20.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:51:20.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:51:20.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:51:20.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:51:21.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:51:21.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-87940-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:21.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:51:21.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:51:21.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T16:51:21.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:51:21.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-17T16:51:21.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-87940-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:21.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-17T16:51:21.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:51:21.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:51:21.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T16:51:21.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:51:21.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T16:51:21.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:51:21.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:51:21.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:51:21.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:51:21.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T16:51:21.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T16:51:21.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:51:21.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:51:21.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:51:21.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:51:21.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:51:21.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:51:21.808 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:51:21.808 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:51:21.808 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:51:21.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:51:21.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:51:21.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:51:21.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:51:21.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:51:21.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:51:21.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:51:21.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:51:21.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:51:21.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:51:21.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:51:21.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T16:51:21.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T16:51:21.809 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-09-17T16:51:21.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:51:21.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-17T16:51:21.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T16:51:21.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:51:21.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:51:21.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:51:21.836 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:21.834+0000 7f64da2b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:51:21.915 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:21.913+0000 7f64da2b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:51:21.916 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:21.914+0000 7f64da2b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:51:22.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T16:51:22.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:51:22.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:51:22.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:51:22.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:51:22.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:22.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:51:22.303 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:51:22.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:22.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:51:22.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:51:22.761 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:22.759+0000 7f64da2b6640 -1 Falling back to public interface 2024-09-17T16:51:23.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:51:23.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:23.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:51:23.711 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:51:23.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:23.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:51:24.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:51:25.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:51:25.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:25.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:51:25.104 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:51:25.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:25.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:51:25.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:51:25.794 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:25.792+0000 7f64da2b6640 -1 osd.3 77 log_to_monitors true 2024-09-17T16:51:26.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:51:26.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:26.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:51:26.498 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:51:26.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:26.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:51:27.094 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [62,77) [v2:127.0.0.1:6826/1577783329,v1:127.0.0.1:6827/1577783329] [v2:127.0.0.1:6828/1577783329,v1:127.0.0.1:6829/1577783329] exists,up 52a64aab-eb89-4359-b407-891acba49384 2024-09-17T16:51:27.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:51:27.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:51:27.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:51:27.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:51:27.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:51:27.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:51:27.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:51:27.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:51:27.095 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:51:27.095 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:51:27.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:51:27.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:51:27.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:51:27.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:51:27.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:51:27.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:51:27.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:51:27.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:51:27.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:51:27.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:51:27.673 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:51:27.673 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:51:27.673 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:51:27.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:51:27.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:27.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:51:27.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803831 2024-09-17T16:51:27.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803831 2024-09-17T16:51:27.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831' 2024-09-17T16:51:27.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:27.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:51:27.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645320 2024-09-17T16:51:27.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645320 2024-09-17T16:51:27.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-309237645320' 2024-09-17T16:51:27.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:27.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:51:28.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378666 2024-09-17T16:51:28.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378666 2024-09-17T16:51:28.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-309237645320 2-81604378666' 2024-09-17T16:51:28.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:28.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:51:28.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318275 2024-09-17T16:51:28.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318275 2024-09-17T16:51:28.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-309237645320 2-81604378666 3-352187318275' 2024-09-17T16:51:28.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:28.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803831 2024-09-17T16:51:28.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:28.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:51:28.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803831 2024-09-17T16:51:28.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:28.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803831 2024-09-17T16:51:28.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803831' 2024-09-17T16:51:28.190 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803831 2024-09-17T16:51:28.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:51:28.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803831 -lt 25769803831 2024-09-17T16:51:28.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:28.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-309237645320 2024-09-17T16:51:28.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:28.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:51:28.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-309237645320 2024-09-17T16:51:28.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:28.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645320 2024-09-17T16:51:28.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 309237645320' 2024-09-17T16:51:28.621 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 309237645320 2024-09-17T16:51:28.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:51:29.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645318 -lt 309237645320 2024-09-17T16:51:29.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:51:30.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:51:30.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:51:30.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645320 -lt 309237645320 2024-09-17T16:51:30.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:30.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378666 2024-09-17T16:51:30.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:30.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:51:30.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378666 2024-09-17T16:51:30.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:30.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378666 2024-09-17T16:51:30.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378666' 2024-09-17T16:51:30.492 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 81604378666 2024-09-17T16:51:30.492 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:51:30.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378667 -lt 81604378666 2024-09-17T16:51:30.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:30.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-352187318275 2024-09-17T16:51:30.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:30.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:51:30.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-352187318275 2024-09-17T16:51:30.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:30.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318275 2024-09-17T16:51:30.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 352187318275' 2024-09-17T16:51:30.918 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 352187318275 2024-09-17T16:51:30.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:51:31.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318275 -lt 352187318275 2024-09-17T16:51:31.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:51:31.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:51:31.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:51:31.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:51:31.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:51:31.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:51:31.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:51:31.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:51:31.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:51:31.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:51:31.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:51:32.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:51:32.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:51:32.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:51:32.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:51:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:51:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:51:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:51:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-17T16:51:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-17T16:51:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-17T16:51:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:256 bytes copied, 3.4888e-05 s, 7.3 MB/s 2024-09-17T16:51:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-87940-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:51:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:51:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T16:51:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:51:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-87940-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:51:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:51:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T16:51:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:51:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-17T16:51:32.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:51:32.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:51:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:51:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:51:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:51:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:51:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-87940-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:51:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:51:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T16:51:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:51:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-17T16:51:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-87940-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-17T16:51:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:51:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:51:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T16:51:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:51:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T16:51:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:51:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:51:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:51:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:51:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T16:51:35.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T16:51:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:51:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:51:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:51:35.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:51:35.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:51:35.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:51:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:51:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:51:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:51:35.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:51:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:51:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:51:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:51:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:51:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:51:35.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:51:35.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:51:35.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:51:35.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:51:35.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:51:35.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T16:51:35.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T16:51:35.247 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-09-17T16:51:35.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:51:35.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-17T16:51:35.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T16:51:35.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:51:35.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:51:35.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:51:35.275 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:35.272+0000 7fe112527640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:51:35.277 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:35.275+0000 7fe112527640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:51:35.279 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:35.276+0000 7fe112527640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:51:35.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T16:51:35.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:51:35.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:51:35.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:51:35.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:51:35.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:35.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:51:35.642 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:51:35.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:35.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:51:36.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:51:36.390 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:36.387+0000 7fe112527640 -1 Falling back to public interface 2024-09-17T16:51:37.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:51:37.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:37.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:51:37.042 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:51:37.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:37.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:51:37.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:51:38.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:51:38.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:38.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:51:38.437 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:51:38.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:38.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:51:38.679 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:38.676+0000 7fe112527640 -1 osd.3 84 log_to_monitors true 2024-09-17T16:51:38.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:51:39.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:51:39.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:39.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:51:39.883 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:51:39.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:39.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:51:40.374 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [82,84) [v2:127.0.0.1:6826/2528381284,v1:127.0.0.1:6827/2528381284] [v2:127.0.0.1:6828/2528381284,v1:127.0.0.1:6829/2528381284] exists,up 52a64aab-eb89-4359-b407-891acba49384 2024-09-17T16:51:40.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:51:40.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:51:40.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:51:40.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:51:40.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:51:40.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:51:40.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:51:40.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:51:40.375 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:51:40.375 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:51:40.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:51:40.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:51:40.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:51:40.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:51:40.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:51:40.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:51:40.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:51:40.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:51:40.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:51:40.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:51:40.859 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:51:40.859 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:51:40.859 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:51:40.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:51:40.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:40.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:51:40.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803836 2024-09-17T16:51:40.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803836 2024-09-17T16:51:40.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803836' 2024-09-17T16:51:40.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:40.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:51:41.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645324 2024-09-17T16:51:41.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645324 2024-09-17T16:51:41.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803836 1-309237645324' 2024-09-17T16:51:41.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:41.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:51:41.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378671 2024-09-17T16:51:41.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378671 2024-09-17T16:51:41.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803836 1-309237645324 2-81604378671' 2024-09-17T16:51:41.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:41.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:51:41.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122051 2024-09-17T16:51:41.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122051 2024-09-17T16:51:41.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803836 1-309237645324 2-81604378671 3-377957122051' 2024-09-17T16:51:41.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:41.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803836 2024-09-17T16:51:41.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:41.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:51:41.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803836 2024-09-17T16:51:41.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:41.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803836 2024-09-17T16:51:41.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803836' 2024-09-17T16:51:41.356 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803836 2024-09-17T16:51:41.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:51:41.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803834 -lt 25769803836 2024-09-17T16:51:41.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:51:42.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:51:42.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:51:43.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803836 -lt 25769803836 2024-09-17T16:51:43.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:43.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-309237645324 2024-09-17T16:51:43.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:43.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:51:43.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-309237645324 2024-09-17T16:51:43.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:43.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645324 2024-09-17T16:51:43.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 309237645324' 2024-09-17T16:51:43.178 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 309237645324 2024-09-17T16:51:43.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:51:43.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645325 -lt 309237645324 2024-09-17T16:51:43.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:43.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378671 2024-09-17T16:51:43.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:43.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:51:43.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378671 2024-09-17T16:51:43.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:43.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378671 2024-09-17T16:51:43.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378671' 2024-09-17T16:51:43.611 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 81604378671 2024-09-17T16:51:43.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:51:43.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378671 -lt 81604378671 2024-09-17T16:51:43.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:43.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-377957122051 2024-09-17T16:51:43.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:43.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:51:43.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-377957122051 2024-09-17T16:51:43.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:43.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122051 2024-09-17T16:51:43.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 377957122051' 2024-09-17T16:51:43.996 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 377957122051 2024-09-17T16:51:43.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:51:44.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122051 -lt 377957122051 2024-09-17T16:51:44.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:51:44.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:51:44.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:51:44.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:51:44.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:51:44.950 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:51:44.950 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:51:44.950 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:51:44.950 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:51:44.950 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:51:44.950 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:51:45.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:51:45.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:51:45.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:51:45.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:51:45.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:51:45.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:51:45.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:51:45.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-17T16:51:45.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-17T16:51:46.542 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-17T16:51:46.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87940-0-256 2024-09-17T16:51:46.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:51:46.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:51:46.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87940-0-256 2024-09-17T16:51:46.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:51:46.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:51:46.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-87940-0-256 td/test-erasure-eio/COPY 2024-09-17T16:51:46.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:51:46.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:51:46.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 0 + 1 2024-09-17T16:51:46.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=1 2024-09-17T16:51:46.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-87940-0-256 td/test-erasure-eio 1 256 add 2024-09-17T16:51:46.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-87940-0-256 2024-09-17T16:51:46.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-17T16:51:46.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-17T16:51:46.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-17T16:51:46.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-17T16:51:46.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-17T16:51:46.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2024-09-17T16:51:46.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-17T16:51:46.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2024-09-17T16:51:46.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-17T16:51:46.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-87940-0-256 2024-09-17T16:51:46.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:51:46.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-87940-0-256 2024-09-17T16:51:46.757 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-87940-0-256 2024-09-17T16:51:46.757 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:51:47.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:51:47.162 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:51:47.163 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:51:47.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T16:51:47.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-17T16:51:47.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-17T16:51:47.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-17T16:51:47.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-17T16:51:47.864 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-17T16:51:47.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2024-09-17T16:51:47.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87940-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:47.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:51:47.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:51:47.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T16:51:47.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:51:47.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87940-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:47.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:51:47.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:51:47.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T16:51:47.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:51:47.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-17T16:51:47.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:51:47.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:51:47.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:51:47.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:51:47.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:51:48.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:51:48.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87940-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:48.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:51:48.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:51:48.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T16:51:48.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:51:48.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-17T16:51:48.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87940-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:51:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-17T16:51:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:51:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:51:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T16:51:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:51:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:51:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:51:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:51:48.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:51:48.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:51:48.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:51:48.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:51:48.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:51:48.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:51:48.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:51:48.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:51:48.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:51:48.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:51:48.904 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:51:48.904 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:51:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:51:48.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:51:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:51:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:51:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:51:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:51:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:51:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:51:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:51:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:51:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:51:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:51:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:51:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T16:51:48.905 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-17T16:51:48.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:51:48.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-17T16:51:48.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T16:51:48.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:51:48.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:51:48.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:51:48.932 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:48.931+0000 7f32ed41a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:51:49.082 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:49.080+0000 7f32ed41a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:51:49.083 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:49.081+0000 7f32ed41a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:51:49.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T16:51:49.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:51:49.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:51:49.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:51:49.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:51:49.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:49.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:51:49.429 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:51:49.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:49.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:51:49.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:51:50.674 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:50.672+0000 7f32ed41a640 -1 Falling back to public interface 2024-09-17T16:51:50.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:51:50.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:50.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:51:50.856 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:51:50.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:50.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:51:51.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:51:52.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:51:52.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:52.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:51:52.282 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:51:52.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:52.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:51:52.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:51:53.704 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:51:53.702+0000 7f32ed41a640 -1 osd.1 93 log_to_monitors true 2024-09-17T16:51:53.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:51:53.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:53.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:51:53.718 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:51:53.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:53.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:51:54.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:51:55.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:51:55.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:51:55.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:51:55.205 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:51:55.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:51:55.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:51:55.757 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 97 up_thru 97 down_at 94 last_clean_interval [72,93) [v2:127.0.0.1:6810/2974995759,v1:127.0.0.1:6811/2974995759] [v2:127.0.0.1:6812/2974995759,v1:127.0.0.1:6813/2974995759] exists,up 792c342f-ff50-45e7-ba85-c4d65b739cb4 2024-09-17T16:51:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:51:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:51:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:51:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:51:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:51:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:51:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:51:55.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:51:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:51:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:51:55.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:51:55.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:51:55.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:51:55.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:51:55.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:51:55.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:51:55.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:51:55.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:51:55.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:51:56.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:51:56.452 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:51:56.452 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:51:56.452 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:51:56.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:51:56.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:56.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:51:56.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803841 2024-09-17T16:51:56.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803841 2024-09-17T16:51:56.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841' 2024-09-17T16:51:56.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:56.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:51:56.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=416611827715 2024-09-17T16:51:56.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 416611827715 2024-09-17T16:51:56.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841 1-416611827715' 2024-09-17T16:51:56.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:56.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:51:56.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378676 2024-09-17T16:51:56.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378676 2024-09-17T16:51:56.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841 1-416611827715 2-81604378676' 2024-09-17T16:51:56.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:51:56.792 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:51:56.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122056 2024-09-17T16:51:56.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122056 2024-09-17T16:51:56.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841 1-416611827715 2-81604378676 3-377957122056' 2024-09-17T16:51:56.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:56.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803841 2024-09-17T16:51:56.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:56.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:51:56.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803841 2024-09-17T16:51:56.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:56.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803841 2024-09-17T16:51:56.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803841' 2024-09-17T16:51:56.908 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803841 2024-09-17T16:51:56.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:51:57.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803839 -lt 25769803841 2024-09-17T16:51:57.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:51:58.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:51:58.338 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:51:58.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803841 -lt 25769803841 2024-09-17T16:51:58.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:58.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-416611827715 2024-09-17T16:51:58.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:58.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:51:58.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-416611827715 2024-09-17T16:51:58.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:58.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=416611827715 2024-09-17T16:51:58.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 416611827715' 2024-09-17T16:51:58.758 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 416611827715 2024-09-17T16:51:58.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:51:59.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 416611827715 -lt 416611827715 2024-09-17T16:51:59.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:59.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378676 2024-09-17T16:51:59.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:59.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:51:59.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378676 2024-09-17T16:51:59.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:59.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378676 2024-09-17T16:51:59.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378676' 2024-09-17T16:51:59.209 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 81604378676 2024-09-17T16:51:59.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:51:59.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378676 -lt 81604378676 2024-09-17T16:51:59.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:51:59.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-377957122056 2024-09-17T16:51:59.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:51:59.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:51:59.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-377957122056 2024-09-17T16:51:59.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:51:59.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122056 2024-09-17T16:51:59.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 377957122056' 2024-09-17T16:51:59.600 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 377957122056 2024-09-17T16:51:59.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:51:59.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122057 -lt 377957122056 2024-09-17T16:51:59.990 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:51:59.990 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:51:59.990 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:52:00.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:52:00.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:52:00.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:52:00.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:52:00.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:52:00.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:52:00.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:52:00.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:52:00.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:52:00.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:52:00.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:52:00.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:52:01.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:52:01.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:52:01.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:52:01.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-17T16:52:01.454 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-17T16:52:01.454 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-17T16:52:01.454 INFO:tasks.workunit.client.0.smithi162.stderr:256 bytes copied, 3.3335e-05 s, 7.7 MB/s 2024-09-17T16:52:01.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87940-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:52:01.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:52:01.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:52:01.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T16:52:01.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:52:01.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87940-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:52:01.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:52:01.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:52:01.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T16:52:01.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:52:01.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-17T16:52:01.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:52:01.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:52:01.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:52:01.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:52:01.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:52:01.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:52:01.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87940-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:52:01.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:52:01.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:52:01.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T16:52:01.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:52:01.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-17T16:52:01.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87940-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:52:04.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-17T16:52:04.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:52:04.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:52:04.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T16:52:04.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:52:04.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:52:04.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:52:04.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:52:04.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:52:04.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:52:04.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:52:04.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:52:04.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:52:04.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:52:04.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:52:04.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:52:04.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:52:04.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:52:04.269 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:52:04.269 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:52:04.269 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:52:04.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:52:04.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:52:04.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:52:04.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:52:04.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:52:04.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:52:04.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:52:04.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:52:04.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:52:04.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:52:04.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:52:04.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:52:04.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T16:52:04.271 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-17T16:52:04.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:52:04.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-17T16:52:04.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T16:52:04.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:52:04.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:52:04.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:52:04.299 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:04.297+0000 7f53e04e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:52:04.301 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:04.299+0000 7f53e04e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:52:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:04.301+0000 7f53e04e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:52:04.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T16:52:04.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:52:04.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:52:04.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:52:04.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:52:04.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:04.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:52:04.702 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:52:04.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:04.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:52:05.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:05.656 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:05.654+0000 7f53e04e0640 -1 Falling back to public interface 2024-09-17T16:52:06.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:06.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:06.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:52:06.091 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:52:06.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:06.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:52:06.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:07.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:07.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:07.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:52:07.506 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:52:07.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:07.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:52:07.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:08.545 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:08.543+0000 7f53e04e0640 -1 osd.1 99 log_to_monitors true 2024-09-17T16:52:08.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:08.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:08.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:52:08.956 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:52:08.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:08.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:52:09.597 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 104 up_thru 104 down_at 100 last_clean_interval [97,99) [v2:127.0.0.1:6810/3622743178,v1:127.0.0.1:6811/3622743178] [v2:127.0.0.1:6812/3622743178,v1:127.0.0.1:6813/3622743178] exists,up 792c342f-ff50-45e7-ba85-c4d65b739cb4 2024-09-17T16:52:09.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:52:09.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:52:09.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:52:09.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:52:09.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:52:09.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:52:09.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:52:09.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:52:09.598 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:52:09.598 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:52:09.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:52:09.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:52:09.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:52:09.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:52:09.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:52:09.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:52:09.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:52:09.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:52:09.687 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:52:10.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:52:10.311 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:52:10.311 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:52:10.311 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:52:10.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:52:10.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:52:10.311 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:52:10.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803846 2024-09-17T16:52:10.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803846 2024-09-17T16:52:10.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846' 2024-09-17T16:52:10.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:52:10.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:52:10.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=446676598787 2024-09-17T16:52:10.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 446676598787 2024-09-17T16:52:10.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846 1-446676598787' 2024-09-17T16:52:10.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:52:10.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:52:10.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378681 2024-09-17T16:52:10.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378681 2024-09-17T16:52:10.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846 1-446676598787 2-81604378681' 2024-09-17T16:52:10.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:52:10.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:52:10.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122061 2024-09-17T16:52:10.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122061 2024-09-17T16:52:10.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846 1-446676598787 2-81604378681 3-377957122061' 2024-09-17T16:52:10.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:52:10.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803846 2024-09-17T16:52:10.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:52:10.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:52:10.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803846 2024-09-17T16:52:10.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:52:10.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803846 2024-09-17T16:52:10.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803846' 2024-09-17T16:52:10.920 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803846 2024-09-17T16:52:10.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:52:11.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803844 -lt 25769803846 2024-09-17T16:52:11.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:52:12.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:52:12.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:52:12.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803846 -lt 25769803846 2024-09-17T16:52:12.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:52:12.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-446676598787 2024-09-17T16:52:12.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:52:12.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:52:12.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-446676598787 2024-09-17T16:52:12.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:52:12.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=446676598787 2024-09-17T16:52:12.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 446676598787' 2024-09-17T16:52:12.760 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 446676598787 2024-09-17T16:52:12.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:52:13.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 446676598787 -lt 446676598787 2024-09-17T16:52:13.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:52:13.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378681 2024-09-17T16:52:13.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:52:13.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:52:13.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378681 2024-09-17T16:52:13.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:52:13.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378681 2024-09-17T16:52:13.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378681' 2024-09-17T16:52:13.152 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 81604378681 2024-09-17T16:52:13.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:52:13.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378681 -lt 81604378681 2024-09-17T16:52:13.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:52:13.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-377957122061 2024-09-17T16:52:13.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:52:13.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:52:13.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-377957122061 2024-09-17T16:52:13.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:52:13.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122061 2024-09-17T16:52:13.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 377957122061' 2024-09-17T16:52:13.579 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 377957122061 2024-09-17T16:52:13.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:52:13.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122061 -lt 377957122061 2024-09-17T16:52:13.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:52:13.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:52:13.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:52:14.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:52:14.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:52:14.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:52:14.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:52:14.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:52:14.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:52:14.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:52:14.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:52:14.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:52:14.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:52:14.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:52:14.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:52:15.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:52:15.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:52:15.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:52:15.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-17T16:52:15.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-17T16:52:16.119 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-17T16:52:16.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 0 -eq 1 -a 1 -eq 2 ']' 2024-09-17T16:52:16.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87940-0-256 fail 2024-09-17T16:52:16.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:52:16.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:52:16.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87940-0-256 2024-09-17T16:52:16.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-17T16:52:16.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-17T16:52:16.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-87940-0-256 td/test-erasure-eio/COPY 2024-09-17T16:52:16.295 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj-size-87940-0-256: (5) Input/output error 2024-09-17T16:52:16.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-17T16:52:16.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-17T16:52:16.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:332: TEST_rados_get_bad_size_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-17T16:52:16.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:52:16.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-17T16:52:16.923 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-17T16:52:16.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-17T16:52:17.449 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-17T16:52:17.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-17T16:52:17.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:52:17.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:52:17.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:52:17.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:52:17.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:52:17.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:52:17.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:52:17.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:52:17.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:52:17.571 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:52:17.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:52:17.571 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:52:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:52:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:52:17.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:52:17.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:52:17.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:52:17.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:52:17.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:52:17.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:52:17.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:52:17.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:52:17.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:52:17.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:52:17.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:52:17.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:52:17.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:52:17.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:52:17.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-17T16:52:17.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-17T16:52:17.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-17T16:52:17.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-17T16:52:17.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:52:17.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:52:17.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:52:17.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:52:17.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:52:17.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:52:17.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:52:17.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:52:17.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:52:17.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:52:17.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:52:17.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:52:17.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:52:17.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:52:17.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:52:17.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:52:17.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:52:17.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:52:17.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:52:17.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:52:17.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:52:17.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:52:17.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:52:17.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:52:17.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:52:17.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:52:17.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:52:17.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:52:17.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-17T16:52:17.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:52:17.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:52:17.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:52:17.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.87940 2024-09-17T16:52:17.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:52:17.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T16:52:17.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:52:17.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-17T16:52:17.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-17T16:52:17.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-17T16:52:17.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:52:17.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:52:17.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:52:17.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-17T16:52:17.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-17T16:52:18.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:52:18.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:52:18.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:52:18.085 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:52:18.085 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:52:18.085 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:52:18.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:52:18.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T16:52:18.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:52:18.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:52:18.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:52:18.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:52:18.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:52:18.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:52:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:52:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:52:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:52:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:52:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:52:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:52:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:52:18.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:52:18.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get fsid 2024-09-17T16:52:18.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:52:18.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:52:18.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:52:18.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:52:18.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:52:18.362 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:52:18.363 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:52:18.363 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:52:18.363 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:52:18.363 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:52:18.363 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:52:18.363 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:52:18.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:52:18.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get mon_host 2024-09-17T16:52:18.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-17T16:52:18.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-17T16:52:18.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:52:18.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:52:18.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:52:18.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-17T16:52:18.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:52:19.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:52:19.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:52:19.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:52:19.290 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:52:19.290 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:52:19.290 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:52:19.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:52:19.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:52:19.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:52:19.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-17T16:52:19.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T16:52:19.609 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T16:52:19.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:52:20.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-17T16:52:20.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:52:20.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:52:20.622 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:52:20.622 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:52:20.622 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:52:20.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:52:20.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-17T16:52:20.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-mon.a.asok log flush 2024-09-17T16:52:20.696 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:52:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-17T16:52:20.706 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:52:18.117+0000 7fb99af2ec80 0 load: jerasure load: lrc 2024-09-17T16:52:20.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_1 td/test-erasure-eio 2024-09-17T16:52:20.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:336: TEST_rados_get_bad_size_shard_1: local dir=td/test-erasure-eio 2024-09-17T16:52:20.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:337: TEST_rados_get_bad_size_shard_1: setup_osds 4 2024-09-17T16:52:20.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-17T16:52:20.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-17T16:52:20.707 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-17T16:52:20.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-17T16:52:20.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:52:20.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-17T16:52:20.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:52:20.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:52:20.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:52:20.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:52:20.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-17T16:52:20.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:52:20.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:52:20.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:52:20.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:52:20.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-17T16:52:20.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-17T16:52:20.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:52:20.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:52:20.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:52:20.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:52:20.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:52:20.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:52:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:52:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:52:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:52:20.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:52:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:52:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:52:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:52:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:52:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:52:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:52:20.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:52:20.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:52:20.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:52:20.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:52:20.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:52:20.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:52:20.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-17T16:52:20.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:52:20.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0b33a245-54e5-48df-bc14-416898133f99 2024-09-17T16:52:20.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0b33a245-54e5-48df-bc14-416898133f99' 2024-09-17T16:52:20.719 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 0b33a245-54e5-48df-bc14-416898133f99 2024-09-17T16:52:20.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:52:20.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDEs+lm7p56KxAAR10wUzJSO557PACOB+l9GQ== 2024-09-17T16:52:20.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDEs+lm7p56KxAAR10wUzJSO557PACOB+l9GQ=="}' 2024-09-17T16:52:20.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0b33a245-54e5-48df-bc14-416898133f99 -i td/test-erasure-eio/0/new.json 2024-09-17T16:52:20.985 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:52:20.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-17T16:52:20.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDEs+lm7p56KxAAR10wUzJSO557PACOB+l9GQ== --osd-uuid 0b33a245-54e5-48df-bc14-416898133f99 2024-09-17T16:52:21.025 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:21.023+0000 7f3655f02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:52:21.054 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:21.052+0000 7f3655f02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:52:21.055 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:21.053+0000 7f3655f02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:52:21.056 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:21.054+0000 7f3655f02640 -1 bdev(0x5596d7f55c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:52:21.056 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:21.054+0000 7f3655f02640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-17T16:52:26.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-17T16:52:26.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:52:26.665 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:52:26.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:52:26.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:52:27.208 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:52:27.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:52:27.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:52:27.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:52:27.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:52:27.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:52:27.236 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:27.233+0000 7fd10a818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:52:27.239 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:27.236+0000 7fd10a818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:52:27.240 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:27.237+0000 7fd10a818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:52:27.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:52:27.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:52:27.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:52:27.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:52:27.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:52:27.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:27.637 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:52:27.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:52:27.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:27.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:28.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:28.839 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:28.836+0000 7fd10a818640 -1 Falling back to public interface 2024-09-17T16:52:29.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:29.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:29.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:52:29.098 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:52:29.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:29.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:29.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:30.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:30.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:30.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:52:30.532 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:52:30.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:30.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:30.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:31.516 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:31.513+0000 7fd10a818640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:52:31.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:31.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:31.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:52:31.982 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:52:31.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:31.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:32.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:33.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:33.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:33.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:52:33.488 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:52:33.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:33.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:33.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:34.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:34.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:34.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:52:34.943 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:52:34.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:34.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:35.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:36.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:36.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:36.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:52:36.357 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:52:36.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:36.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:36.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:37.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:37.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:37.762 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:52:37.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:52:37.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:37.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:38.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:39.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:39.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:39.245 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:52:39.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:52:39.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:39.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:39.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:40.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:40.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:40.659 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:52:40.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:52:40.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:40.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:41.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:42.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:42.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:42.236 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:52:42.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:52:42.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:42.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:42.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:43.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:43.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:43.865 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:52:43.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:52:43.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:43.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:44.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:45.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:45.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:45.878 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:52:45.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:52:45.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:45.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:46.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:47.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:47.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:47.829 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:52:47.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:52:47.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:47.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:49.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:50.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:50.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:50.144 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:52:50.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:52:50.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:50.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:50.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:51.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:51.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:51.607 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:52:51.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:52:51.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:51.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:52.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:53.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:53.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:53.962 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:52:53.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:52:53.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:53.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:54.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:52:55.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:52:55.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:52:55.512 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:52:55.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:52:55.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:52:55.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:52:55.912 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3228531998,v1:127.0.0.1:6803/3228531998] [v2:127.0.0.1:6804/3228531998,v1:127.0.0.1:6805/3228531998] exists,up 0b33a245-54e5-48df-bc14-416898133f99 2024-09-17T16:52:55.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:52:55.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:52:55.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:52:55.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:52:55.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-17T16:52:55.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:52:55.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:52:55.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T16:52:55.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:52:55.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:52:55.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:52:55.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:52:55.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:52:55.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:52:55.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:52:55.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:52:55.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:52:55.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:52:55.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:52:55.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:52:55.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:52:55.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:52:55.921 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:52:55.921 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:52:55.921 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:52:55.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:52:55.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:52:55.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:52:55.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:52:55.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:52:55.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:52:55.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:52:55.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:52:55.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:52:55.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:52:55.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:52:55.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:52:55.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:52:55.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:52:55.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:52:55.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=94215679-4d44-47a5-8a6a-ab5394377388 2024-09-17T16:52:55.923 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 94215679-4d44-47a5-8a6a-ab5394377388 2024-09-17T16:52:55.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 94215679-4d44-47a5-8a6a-ab5394377388' 2024-09-17T16:52:55.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:52:55.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDns+lmTaylNxAAMLwQsXxfYJWWX+z0Ly9pLQ== 2024-09-17T16:52:55.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDns+lmTaylNxAAMLwQsXxfYJWWX+z0Ly9pLQ=="}' 2024-09-17T16:52:55.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 94215679-4d44-47a5-8a6a-ab5394377388 -i td/test-erasure-eio/1/new.json 2024-09-17T16:52:56.393 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:52:56.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-17T16:52:56.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDns+lmTaylNxAAMLwQsXxfYJWWX+z0Ly9pLQ== --osd-uuid 94215679-4d44-47a5-8a6a-ab5394377388 2024-09-17T16:52:56.435 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:56.433+0000 7f64f28fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:52:56.437 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:56.435+0000 7f64f28fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:52:56.437 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:56.436+0000 7f64f28fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:52:56.438 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:56.436+0000 7f64f28fe640 -1 bdev(0x55bbd5ce9c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-17T16:52:56.438 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:52:56.436+0000 7f64f28fe640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-17T16:53:01.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-17T16:53:01.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:53:01.869 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T16:53:01.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T16:53:01.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:53:02.406 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T16:53:02.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T16:53:02.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:53:02.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:53:02.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:53:02.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:53:02.434 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:53:02.432+0000 7f777a415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:53:02.445 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:53:02.443+0000 7f777a415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:53:02.446 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:53:02.444+0000 7f777a415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:53:02.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T16:53:02.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:53:02.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:53:02.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:53:02.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:53:02.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:02.852 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:53:02.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:53:02.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:02.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:03.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:03.299 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:53:03.297+0000 7f777a415640 -1 Falling back to public interface 2024-09-17T16:53:04.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:04.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:04.249 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:53:04.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:53:04.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:04.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:04.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:05.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:05.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:05.710 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:53:05.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:53:05.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:05.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:06.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:06.322 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:53:06.320+0000 7f777a415640 -1 osd.1 0 log_to_monitors true 2024-09-17T16:53:07.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:07.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:07.115 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:53:07.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:53:07.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:07.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:07.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:08.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:08.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:08.604 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:53:08.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:53:08.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:08.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:09.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:10.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:10.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:10.006 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:53:10.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:53:10.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:10.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:10.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:11.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:11.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:11.493 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:53:11.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:53:11.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:11.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:11.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:12.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:12.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:12.938 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:53:12.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:53:12.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:12.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:13.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:14.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:14.463 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:53:14.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:14.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:53:14.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:14.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:15.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:15.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:15.863 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:53:15.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:53:15.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:15.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:16.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:17.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:17.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:17.323 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:53:17.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:53:17.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:17.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:17.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:18.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:18.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:18.863 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:53:18.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:53:18.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:18.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:19.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:20.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:20.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:20.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:53:20.875 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:53:20.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:20.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:21.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:22.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:22.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:22.610 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:53:22.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:53:22.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:22.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:23.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:24.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:24.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:24.438 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:53:24.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:53:24.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:24.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:25.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:26.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:26.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:26.273 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:53:26.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:53:26.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:26.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:26.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:27.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:27.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:27.949 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:53:27.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:53:27.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:27.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:29.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:30.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:30.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:30.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:53:30.021 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:53:30.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:30.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:53:30.427 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2725532114,v1:127.0.0.1:6811/2725532114] [v2:127.0.0.1:6812/2725532114,v1:127.0.0.1:6813/2725532114] exists,up 94215679-4d44-47a5-8a6a-ab5394377388 2024-09-17T16:53:30.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:53:30.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:53:30.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:53:30.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:53:30.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-17T16:53:30.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:53:30.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:53:30.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T16:53:30.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:53:30.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:53:30.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:53:30.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:53:30.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:53:30.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:53:30.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:53:30.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:53:30.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:53:30.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:53:30.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:53:30.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:53:30.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:53:30.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:53:30.433 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:53:30.433 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:30.433 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:53:30.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:53:30.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:53:30.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:53:30.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:53:30.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:53:30.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:53:30.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:53:30.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:53:30.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:53:30.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:53:30.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:53:30.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:53:30.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:53:30.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:53:30.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:53:30.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0be25c09-9bd9-40c5-9fa5-fc87d2b8d363 2024-09-17T16:53:30.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 0be25c09-9bd9-40c5-9fa5-fc87d2b8d363' 2024-09-17T16:53:30.435 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 0be25c09-9bd9-40c5-9fa5-fc87d2b8d363 2024-09-17T16:53:30.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:53:30.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAKtOlmAE63GhAA56Ifwp1YWp4v+CpoU6vEeA== 2024-09-17T16:53:30.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAKtOlmAE63GhAA56Ifwp1YWp4v+CpoU6vEeA=="}' 2024-09-17T16:53:30.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0be25c09-9bd9-40c5-9fa5-fc87d2b8d363 -i td/test-erasure-eio/2/new.json 2024-09-17T16:53:31.045 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:53:31.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-17T16:53:31.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAKtOlmAE63GhAA56Ifwp1YWp4v+CpoU6vEeA== --osd-uuid 0be25c09-9bd9-40c5-9fa5-fc87d2b8d363 2024-09-17T16:53:31.087 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:53:31.085+0000 7f328de85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:53:31.089 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:53:31.087+0000 7f328de85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:53:31.090 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:53:31.088+0000 7f328de85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:53:31.090 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:53:31.088+0000 7f328de85640 -1 bdev(0x56247b5a3c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-17T16:53:31.090 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:53:31.088+0000 7f328de85640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-17T16:53:37.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-17T16:53:37.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:53:37.056 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T16:53:37.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T16:53:37.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:53:37.617 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T16:53:37.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T16:53:37.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:53:37.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:53:37.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:53:37.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:53:37.644 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:53:37.642+0000 7f209ffb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:53:37.676 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:53:37.674+0000 7f209ffb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:53:37.677 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:53:37.675+0000 7f209ffb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:53:38.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T16:53:38.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:53:38.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:53:38.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:53:38.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:53:38.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:38.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:53:38.002 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:53:38.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:38.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:53:38.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:39.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:39.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:39.432 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:53:39.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:53:39.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:39.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:53:39.524 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:53:39.522+0000 7f209ffb2640 -1 Falling back to public interface 2024-09-17T16:53:39.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:40.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:40.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:40.866 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:53:40.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:53:40.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:40.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:53:41.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:42.170 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:53:42.168+0000 7f209ffb2640 -1 osd.2 0 log_to_monitors true 2024-09-17T16:53:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:53:42.265 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:53:42.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:42.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:53:42.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:43.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:43.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:43.833 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:53:43.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:53:43.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:43.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:53:44.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:45.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:45.298 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:53:45.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:45.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:53:45.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:45.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:53:45.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:46.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:46.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:46.836 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:53:46.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:53:46.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:46.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:53:47.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:48.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:48.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:48.307 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:53:48.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:53:48.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:48.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:53:48.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:49.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:49.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:49.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:53:49.715 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:53:49.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:49.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:53:50.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:51.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:51.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:51.153 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:53:51.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:53:51.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:51.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:53:51.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:52.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:52.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:52.602 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:53:52.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:53:52.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:52.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:53:53.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:54.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:54.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:54.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:53:54.107 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:53:54.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:54.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:53:54.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:55.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:55.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:55.503 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:53:55.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:53:55.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:55.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:53:56.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:57.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:57.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:57.365 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:53:57.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:53:57.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:57.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:53:57.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:53:58.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:53:58.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:53:58.816 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:53:58.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:53:58.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:53:58.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:53:59.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:00.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:00.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:00.454 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:54:00.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:54:00.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:00.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:54:01.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:02.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:02.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:02.257 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:54:02.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:54:02.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:02.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:54:03.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:04.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:04.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:04.075 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:54:04.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:54:04.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:04.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:54:04.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:05.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:05.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:05.931 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:54:05.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:54:05.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:05.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:54:06.381 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2114837865,v1:127.0.0.1:6819/2114837865] [v2:127.0.0.1:6820/2114837865,v1:127.0.0.1:6821/2114837865] exists,up 0be25c09-9bd9-40c5-9fa5-fc87d2b8d363 2024-09-17T16:54:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:54:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:54:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:54:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:54:06.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-17T16:54:06.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:54:06.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:54:06.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T16:54:06.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:54:06.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T16:54:06.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:54:06.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:54:06.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:54:06.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:54:06.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T16:54:06.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T16:54:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:54:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:54:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:54:06.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:54:06.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:54:06.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:54:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:54:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:54:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:54:06.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:54:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:54:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:54:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:54:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:54:06.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:54:06.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:54:06.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:54:06.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:54:06.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:54:06.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:54:06.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:54:06.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:54:06.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T16:54:06.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:54:06.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=89b32915-52f3-4254-ae8b-28f5a451f432 2024-09-17T16:54:06.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 89b32915-52f3-4254-ae8b-28f5a451f432' 2024-09-17T16:54:06.389 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 89b32915-52f3-4254-ae8b-28f5a451f432 2024-09-17T16:54:06.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:54:06.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAutOlmzHv0FxAAUedWWKWmogFpb2grhx+7UQ== 2024-09-17T16:54:06.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAutOlmzHv0FxAAUedWWKWmogFpb2grhx+7UQ=="}' 2024-09-17T16:54:06.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 89b32915-52f3-4254-ae8b-28f5a451f432 -i td/test-erasure-eio/3/new.json 2024-09-17T16:54:07.027 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:54:07.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-17T16:54:07.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAutOlmzHv0FxAAUedWWKWmogFpb2grhx+7UQ== --osd-uuid 89b32915-52f3-4254-ae8b-28f5a451f432 2024-09-17T16:54:07.068 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:54:07.065+0000 7f1f14c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:07.069 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:54:07.066+0000 7f1f14c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:07.070 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:54:07.067+0000 7f1f14c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:07.071 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:54:07.068+0000 7f1f14c0b640 -1 bdev(0x559375fdbc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-17T16:54:07.071 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:54:07.068+0000 7f1f14c0b640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-17T16:54:13.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-17T16:54:13.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:54:13.769 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-17T16:54:13.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T16:54:13.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:54:14.312 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T16:54:14.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T16:54:14.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:54:14.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:54:14.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:54:14.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:54:14.339 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:54:14.337+0000 7fd68fa0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:14.352 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:54:14.349+0000 7fd68fa0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:14.353 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:54:14.351+0000 7fd68fa0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:14.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T16:54:14.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:54:14.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:54:14.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:54:14.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:54:14.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:14.723 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:54:14.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:54:14.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:14.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:15.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:15.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:54:15.938+0000 7fd68fa0d640 -1 Falling back to public interface 2024-09-17T16:54:16.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:16.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:16.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:54:16.177 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:54:16.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:16.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:16.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:17.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:17.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:17.646 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:54:17.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:54:17.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:17.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:18.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:18.583 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:54:18.580+0000 7fd68fa0d640 -1 osd.3 0 log_to_monitors true 2024-09-17T16:54:19.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:19.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:19.053 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:54:19.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:54:19.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:19.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:19.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:20.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:20.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:20.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:54:20.574 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:54:20.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:20.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:21.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:22.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:22.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:22.213 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:54:22.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:54:22.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:22.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:22.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:23.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:23.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:23.645 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:54:23.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:54:23.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:23.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:24.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:25.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:25.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:25.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:54:25.059 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:54:25.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:25.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:25.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:26.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:26.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:26.628 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:54:26.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:54:26.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:26.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:27.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:28.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:28.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:28.039 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:54:28.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:54:28.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:28.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:28.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:29.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:29.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:29.514 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:54:29.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:54:29.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:29.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:29.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:30.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:30.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:30.963 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:54:30.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:54:30.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:30.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:31.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:32.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:32.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:32.508 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:54:32.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:54:32.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:32.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:33.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:34.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:34.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:34.276 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:54:34.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:54:34.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:34.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:35.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:36.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:36.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:36.064 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:54:36.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:54:36.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:36.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:37.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:38.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:38.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:38.011 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:54:38.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:54:38.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:38.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:39.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:40.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:40.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:40.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:54:40.172 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:54:40.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:40.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:41.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:42.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:42.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:42.182 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:54:42.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:54:42.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:42.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:43.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:44.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:44.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:44.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:54:44.056 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:54:44.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:44.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:54:44.603 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 30 up_thru 31 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1730705992,v1:127.0.0.1:6827/1730705992] [v2:127.0.0.1:6828/1730705992,v1:127.0.0.1:6829/1730705992] exists,up 89b32915-52f3-4254-ae8b-28f5a451f432 2024-09-17T16:54:44.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:54:44.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:54:44.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:54:44.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-17T16:54:44.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:54:44.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:54:44.606 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:54:44.607 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:54:44.607 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:54:44.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T16:54:44.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-17T16:54:44.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.0.asok log flush 2024-09-17T16:54:44.684 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:54:44.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-17T16:54:44.695 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:52:28.840+0000 7fd10a818640 0 load: jerasure load: lrc 2024-09-17T16:54:44.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:339: TEST_rados_get_bad_size_shard_1: local poolname=pool-jerasure 2024-09-17T16:54:44.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:340: TEST_rados_get_bad_size_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-17T16:54:44.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:54:44.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-17T16:54:44.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-17T16:54:44.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-17T16:54:44.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-17T16:54:44.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-17T16:54:44.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-17T16:54:45.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-17T16:54:45.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-17T16:54:46.469 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-17T16:54:46.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:54:47.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-17T16:54:47.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:54:47.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:54:47.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:54:47.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:54:47.484 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:54:47.484 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:54:47.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:54:47.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:54:47.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:54:47.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:54:47.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:54:47.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:54:47.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:54:47.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:54:47.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:54:48.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:54:48.082 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:54:48.082 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:54:48.082 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:54:48.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:54:48.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:54:48.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:54:48.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803804 2024-09-17T16:54:48.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803804 2024-09-17T16:54:48.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803804' 2024-09-17T16:54:48.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:54:48.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:54:48.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574871 2024-09-17T16:54:48.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574871 2024-09-17T16:54:48.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803804 1-55834574871' 2024-09-17T16:54:48.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:54:48.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:54:48.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345934 2024-09-17T16:54:48.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345934 2024-09-17T16:54:48.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803804 1-55834574871 2-85899345934' 2024-09-17T16:54:48.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:54:48.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:54:48.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018885 2024-09-17T16:54:48.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018885 2024-09-17T16:54:48.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803804 1-55834574871 2-85899345934 3-128849018885' 2024-09-17T16:54:48.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:54:48.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803804 2024-09-17T16:54:48.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:54:48.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:54:48.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803804 2024-09-17T16:54:48.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:54:48.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803804 2024-09-17T16:54:48.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803804' 2024-09-17T16:54:48.637 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803804 2024-09-17T16:54:48.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:54:49.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803802 -lt 25769803804 2024-09-17T16:54:49.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:54:50.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:54:50.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:54:50.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803804 -lt 25769803804 2024-09-17T16:54:50.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:54:50.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574871 2024-09-17T16:54:50.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:54:50.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:54:50.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574871 2024-09-17T16:54:50.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:54:50.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574871 2024-09-17T16:54:50.474 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574871 2024-09-17T16:54:50.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574871' 2024-09-17T16:54:50.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:54:50.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574871 -lt 55834574871 2024-09-17T16:54:50.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:54:50.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345934 2024-09-17T16:54:50.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:54:50.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:54:50.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345934 2024-09-17T16:54:50.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:54:50.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345934 2024-09-17T16:54:50.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345934' 2024-09-17T16:54:50.909 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 85899345934 2024-09-17T16:54:50.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:54:51.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345934 -lt 85899345934 2024-09-17T16:54:51.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:54:51.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018885 2024-09-17T16:54:51.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:54:51.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:54:51.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018885 2024-09-17T16:54:51.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:54:51.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018885 2024-09-17T16:54:51.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018885' 2024-09-17T16:54:51.335 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 128849018885 2024-09-17T16:54:51.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:54:51.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018886 -lt 128849018885 2024-09-17T16:54:51.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:54:51.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:54:51.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:54:52.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:54:52.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:54:52.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:54:52.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:54:52.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:54:52.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:54:52.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:54:52.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:54:52.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:54:52.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:54:52.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:54:52.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:54:53.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:54:53.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:54:53.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:54:53.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:342: TEST_rados_get_bad_size_shard_1: local shard_id=1 2024-09-17T16:54:53.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:343: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 10 2024-09-17T16:54:53.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-17T16:54:53.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-17T16:54:53.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=1 2024-09-17T16:54:53.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-17T16:54:53.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=10 2024-09-17T16:54:53.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-17T16:54:53.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2024-09-17T16:54:53.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-17T16:54:53.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-87940-1-10 2024-09-17T16:54:53.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-87940-1-10 2024-09-17T16:54:53.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:54:53.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:54:53.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-87940-1-10 2024-09-17T16:54:53.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:54:53.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:54:53.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:54:53.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:54:53.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:54:53.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:54:53.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:54:53.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:54:53.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-87940-1-10 td/test-erasure-eio/ORIGINAL 2024-09-17T16:54:53.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-87940-1-10 td/test-erasure-eio 1 10 set 2024-09-17T16:54:53.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-87940-1-10 2024-09-17T16:54:53.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-17T16:54:53.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-17T16:54:53.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-17T16:54:53.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-17T16:54:53.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-17T16:54:53.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2024-09-17T16:54:53.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-17T16:54:53.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-17T16:54:53.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-17T16:54:53.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-87940-1-10 2024-09-17T16:54:53.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:54:53.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-87940-1-10 2024-09-17T16:54:53.329 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-87940-1-10 2024-09-17T16:54:53.329 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:54:53.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:54:53.783 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:54:53.783 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:54:53.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T16:54:53.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-17T16:54:53.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-17T16:54:53.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-17T16:54:53.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-17T16:54:54.395 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-17T16:54:54.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-17T16:54:54.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2024-09-17T16:54:54.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-17T16:54:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-17T16:54:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-17T16:54:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:10 bytes copied, 4.6323e-05 s, 216 kB/s 2024-09-17T16:54:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87940-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:54:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:54:54.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:54:54.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T16:54:54.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:54:54.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87940-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:54:54.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:54:54.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:54:54.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T16:54:54.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:54:54.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-17T16:54:54.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:54:54.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:54:54.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:54:54.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:54:54.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:54:55.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:54:55.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87940-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:54:55.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:54:55.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:54:55.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T16:54:55.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:54:55.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-17T16:54:55.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87940-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:54:57.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-17T16:54:57.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:54:57.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:54:57.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T16:54:57.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:54:57.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:54:57.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:54:57.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:54:57.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:54:57.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:54:57.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:54:57.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:54:57.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:54:57.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:54:57.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:54:57.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:54:57.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:54:57.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:54:57.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:54:57.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:54:57.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:54:57.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:54:57.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:54:57.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:54:57.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:54:57.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:54:57.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:54:57.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:54:57.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:54:57.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:54:57.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:54:57.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:54:57.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:54:57.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T16:54:57.752 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-17T16:54:57.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:54:57.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-17T16:54:57.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T16:54:57.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:54:57.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:54:57.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:54:57.779 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:54:57.776+0000 7f78e3e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:57.782 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:54:57.779+0000 7f78e3e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:57.783 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:54:57.780+0000 7f78e3e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:58.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T16:54:58.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:54:58.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:54:58.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:54:58.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:54:58.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:58.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:54:58.167 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:54:58.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:58.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:54:58.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:59.126 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:54:59.124+0000 7f78e3e12640 -1 Falling back to public interface 2024-09-17T16:54:59.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:59.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:59.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:54:59.627 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:54:59.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:59.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:00.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:01.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:01.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:01.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:55:01.055 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:55:01.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:01.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:01.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:01.727 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:01.725+0000 7f78e3e12640 -1 osd.1 41 log_to_monitors true 2024-09-17T16:55:02.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:02.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:02.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:55:02.456 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:55:02.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:02.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:02.924 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [13,41) [v2:127.0.0.1:6810/3636181134,v1:127.0.0.1:6811/3636181134] [v2:127.0.0.1:6812/3636181134,v1:127.0.0.1:6813/3636181134] exists,up 94215679-4d44-47a5-8a6a-ab5394377388 2024-09-17T16:55:02.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:55:02.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:55:02.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:55:02.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:55:02.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:55:02.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:55:02.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:55:02.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:55:02.924 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:55:02.924 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:55:02.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:55:02.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:55:02.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:55:03.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:55:03.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:55:03.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:55:03.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:55:03.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:55:03.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:55:03.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:55:03.426 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:55:03.426 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:55:03.426 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:55:03.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:55:03.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:03.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:55:03.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803809 2024-09-17T16:55:03.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803809 2024-09-17T16:55:03.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809' 2024-09-17T16:55:03.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:03.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:55:03.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528323 2024-09-17T16:55:03.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528323 2024-09-17T16:55:03.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-193273528323' 2024-09-17T16:55:03.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:03.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:55:03.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345939 2024-09-17T16:55:03.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345939 2024-09-17T16:55:03.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-193273528323 2-85899345939' 2024-09-17T16:55:03.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:03.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:55:04.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018890 2024-09-17T16:55:04.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018890 2024-09-17T16:55:04.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-193273528323 2-85899345939 3-128849018890' 2024-09-17T16:55:04.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:04.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803809 2024-09-17T16:55:04.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:04.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:55:04.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803809 2024-09-17T16:55:04.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:04.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803809 2024-09-17T16:55:04.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2024-09-17T16:55:04.039 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803809 2024-09-17T16:55:04.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:55:04.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803807 -lt 25769803809 2024-09-17T16:55:04.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:55:05.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:55:05.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:55:05.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803810 -lt 25769803809 2024-09-17T16:55:05.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:05.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-193273528323 2024-09-17T16:55:05.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:05.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:55:05.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-193273528323 2024-09-17T16:55:05.876 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:05.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528323 2024-09-17T16:55:05.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 193273528323' 2024-09-17T16:55:05.877 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 193273528323 2024-09-17T16:55:05.877 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:55:06.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528323 -lt 193273528323 2024-09-17T16:55:06.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:06.342 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345939 2024-09-17T16:55:06.342 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:06.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:55:06.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345939 2024-09-17T16:55:06.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:06.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345939 2024-09-17T16:55:06.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345939' 2024-09-17T16:55:06.345 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 85899345939 2024-09-17T16:55:06.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:55:06.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345939 -lt 85899345939 2024-09-17T16:55:06.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:06.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018890 2024-09-17T16:55:06.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:06.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:55:06.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018890 2024-09-17T16:55:06.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:06.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018890 2024-09-17T16:55:06.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018890' 2024-09-17T16:55:06.796 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 128849018890 2024-09-17T16:55:06.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:55:07.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018891 -lt 128849018890 2024-09-17T16:55:07.220 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:55:07.220 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:55:07.220 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:55:07.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:55:07.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:55:07.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:55:07.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:55:07.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:55:07.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:55:07.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:55:07.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:55:08.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:55:08.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:55:08.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:55:08.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:55:08.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:55:08.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:55:08.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:55:08.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-17T16:55:08.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-17T16:55:09.311 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-17T16:55:09.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87940-1-10 2024-09-17T16:55:09.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:55:09.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:55:09.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87940-1-10 2024-09-17T16:55:09.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:55:09.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:55:09.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-87940-1-10 td/test-erasure-eio/COPY 2024-09-17T16:55:09.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:55:09.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:55:09.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 1 + 1 2024-09-17T16:55:09.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=2 2024-09-17T16:55:09.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-87940-1-10 td/test-erasure-eio 2 10 set 2024-09-17T16:55:09.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-87940-1-10 2024-09-17T16:55:09.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-17T16:55:09.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-17T16:55:09.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-17T16:55:09.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=2 2024-09-17T16:55:09.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-17T16:55:09.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2024-09-17T16:55:09.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-17T16:55:09.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-17T16:55:09.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-17T16:55:09.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-87940-1-10 2024-09-17T16:55:09.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:55:09.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-87940-1-10 2024-09-17T16:55:09.427 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-87940-1-10 2024-09-17T16:55:09.427 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:55:09.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:55:09.869 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:55:09.869 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:55:09.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T16:55:09.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-17T16:55:09.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-17T16:55:09.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=2 2024-09-17T16:55:09.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-17T16:55:10.491 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-17T16:55:10.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-17T16:55:10.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2024-09-17T16:55:10.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-17T16:55:10.505 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-17T16:55:10.505 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-17T16:55:10.505 INFO:tasks.workunit.client.0.smithi162.stderr:10 bytes copied, 4.5903e-05 s, 218 kB/s 2024-09-17T16:55:10.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-87940-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:55:10.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:55:10.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:55:10.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-17T16:55:10.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:55:10.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-87940-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:55:10.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:55:10.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:55:10.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-17T16:55:10.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:55:10.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-17T16:55:10.506 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:55:10.506 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:55:10.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:55:10.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:55:10.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:55:10.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:55:10.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-87940-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:55:10.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:55:10.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:55:10.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-17T16:55:10.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:55:10.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-17T16:55:10.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-87940-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:55:13.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-17T16:55:13.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:55:13.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:55:13.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T16:55:13.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:55:13.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:55:13.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:55:13.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:55:13.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:55:13.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:55:13.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:55:13.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:55:13.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:55:13.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:55:13.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:55:13.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:55:13.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:55:13.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:55:13.385 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:55:13.385 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:55:13.385 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:55:13.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:55:13.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:55:13.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:55:13.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:55:13.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:55:13.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:55:13.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:55:13.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:55:13.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:55:13.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:55:13.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:55:13.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:55:13.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T16:55:13.386 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-09-17T16:55:13.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:55:13.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-17T16:55:13.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T16:55:13.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:55:13.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:55:13.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:55:13.413 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:13.411+0000 7fac3b9a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:13.424 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:13.421+0000 7fac3b9a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:13.424 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:13.423+0000 7fac3b9a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:13.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T16:55:13.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:55:13.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:55:13.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:55:13.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:55:13.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:13.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:55:13.813 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:55:13.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:13.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:55:14.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:14.778 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:14.776+0000 7fac3b9a8640 -1 Falling back to public interface 2024-09-17T16:55:15.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:15.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:15.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:55:15.267 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:55:15.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:15.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:55:15.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:16.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:16.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:16.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:55:16.665 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:55:16.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:16.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:55:17.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:17.937 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:17.935+0000 7fac3b9a8640 -1 osd.2 51 log_to_monitors true 2024-09-17T16:55:18.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:18.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:18.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:55:18.104 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:55:18.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:18.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:55:18.727 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 56 up_thru 42 down_at 52 last_clean_interval [20,51) [v2:127.0.0.1:6818/2085073824,v1:127.0.0.1:6819/2085073824] [v2:127.0.0.1:6820/2085073824,v1:127.0.0.1:6821/2085073824] exists,up 0be25c09-9bd9-40c5-9fa5-fc87d2b8d363 2024-09-17T16:55:18.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:55:18.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:55:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:55:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:55:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:55:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:55:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:55:18.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:55:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:55:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:55:18.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:55:18.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:55:18.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:55:18.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:55:18.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:55:18.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:55:18.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:55:18.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:55:18.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:55:19.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:55:19.381 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:55:19.381 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:55:19.381 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:55:19.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:55:19.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:19.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:55:19.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803814 2024-09-17T16:55:19.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803814 2024-09-17T16:55:19.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803814' 2024-09-17T16:55:19.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:19.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:55:19.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528328 2024-09-17T16:55:19.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528328 2024-09-17T16:55:19.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803814 1-193273528328' 2024-09-17T16:55:19.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:19.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:55:19.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=240518168579 2024-09-17T16:55:19.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 240518168579 2024-09-17T16:55:19.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803814 1-193273528328 2-240518168579' 2024-09-17T16:55:19.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:19.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:55:20.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018896 2024-09-17T16:55:20.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018896 2024-09-17T16:55:20.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803814 1-193273528328 2-240518168579 3-128849018896' 2024-09-17T16:55:20.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:20.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803814 2024-09-17T16:55:20.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:20.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:55:20.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803814 2024-09-17T16:55:20.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:20.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803814 2024-09-17T16:55:20.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803814' 2024-09-17T16:55:20.074 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803814 2024-09-17T16:55:20.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:55:20.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803812 -lt 25769803814 2024-09-17T16:55:20.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:55:21.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:55:21.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:55:21.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803815 -lt 25769803814 2024-09-17T16:55:21.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:21.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-193273528328 2024-09-17T16:55:21.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:21.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:55:21.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-193273528328 2024-09-17T16:55:21.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:21.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528328 2024-09-17T16:55:21.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 193273528328' 2024-09-17T16:55:21.926 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 193273528328 2024-09-17T16:55:21.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:55:22.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528328 -lt 193273528328 2024-09-17T16:55:22.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:22.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-240518168579 2024-09-17T16:55:22.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:22.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:55:22.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-240518168579 2024-09-17T16:55:22.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:22.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=240518168579 2024-09-17T16:55:22.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 240518168579' 2024-09-17T16:55:22.311 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 240518168579 2024-09-17T16:55:22.311 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:55:22.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 240518168579 -lt 240518168579 2024-09-17T16:55:22.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:22.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018896 2024-09-17T16:55:22.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:22.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:55:22.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018896 2024-09-17T16:55:22.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:22.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018896 2024-09-17T16:55:22.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018896' 2024-09-17T16:55:22.738 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 128849018896 2024-09-17T16:55:22.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:55:23.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018896 -lt 128849018896 2024-09-17T16:55:23.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:55:23.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:55:23.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:55:23.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:55:23.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:55:23.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:55:23.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:55:23.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:55:23.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:55:23.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:55:23.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:55:24.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:55:24.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:55:24.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:55:24.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:55:24.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:55:24.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:55:24.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:55:24.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-17T16:55:24.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-17T16:55:25.269 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-17T16:55:25.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 1 -eq 1 -a 2 -eq 2 ']' 2024-09-17T16:55:25.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87940-1-10 2024-09-17T16:55:25.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:55:25.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:55:25.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87940-1-10 2024-09-17T16:55:25.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:55:25.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:55:25.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-87940-1-10 td/test-erasure-eio/COPY 2024-09-17T16:55:25.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:55:25.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:55:25.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-17T16:55:25.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:344: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 0 2024-09-17T16:55:25.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-17T16:55:25.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-17T16:55:25.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=1 2024-09-17T16:55:25.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-17T16:55:25.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=0 2024-09-17T16:55:25.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-17T16:55:25.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2024-09-17T16:55:25.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-17T16:55:25.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-87940-1-0 2024-09-17T16:55:25.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-87940-1-0 2024-09-17T16:55:25.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:55:25.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:55:25.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-87940-1-0 2024-09-17T16:55:25.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:55:25.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:55:25.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:55:25.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:55:25.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:55:25.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:55:25.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:55:25.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:55:25.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-87940-1-0 td/test-erasure-eio/ORIGINAL 2024-09-17T16:55:25.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-87940-1-0 td/test-erasure-eio 1 0 set 2024-09-17T16:55:25.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-87940-1-0 2024-09-17T16:55:25.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-17T16:55:25.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-17T16:55:25.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-17T16:55:25.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-17T16:55:25.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-17T16:55:25.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2024-09-17T16:55:25.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-17T16:55:25.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-17T16:55:25.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-17T16:55:25.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-87940-1-0 2024-09-17T16:55:25.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:55:25.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-87940-1-0 2024-09-17T16:55:25.618 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-87940-1-0 2024-09-17T16:55:25.618 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:55:26.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:55:26.079 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:55:26.079 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:55:26.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T16:55:26.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-17T16:55:26.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-17T16:55:26.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-17T16:55:26.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-17T16:55:26.793 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-17T16:55:26.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-17T16:55:26.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2024-09-17T16:55:26.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-17T16:55:26.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87940-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:55:26.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:55:26.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:55:26.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T16:55:26.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:55:26.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87940-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:55:26.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:55:26.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:55:26.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T16:55:26.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:55:26.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-17T16:55:26.809 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:55:26.809 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:55:26.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:55:26.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:55:26.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:55:27.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:55:27.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87940-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:55:27.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:55:27.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:55:27.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T16:55:27.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:55:27.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-17T16:55:27.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87940-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:55:29.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-17T16:55:29.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:55:29.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:55:29.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T16:55:29.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:55:29.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:55:29.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:55:29.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:55:29.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:55:29.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:55:29.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:55:29.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:55:29.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:55:29.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:55:29.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:55:29.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:55:29.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:55:29.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:55:29.452 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:55:29.452 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:55:29.452 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:55:29.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:55:29.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:55:29.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:55:29.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:55:29.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:55:29.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:55:29.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:55:29.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:55:29.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:55:29.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:55:29.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:55:29.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:55:29.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T16:55:29.457 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-17T16:55:29.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:55:29.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-17T16:55:29.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T16:55:29.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:55:29.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:55:29.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:55:29.482 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:29.480+0000 7f4cf100d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:29.501 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:29.498+0000 7f4cf100d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:29.502 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:29.500+0000 7f4cf100d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:29.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T16:55:29.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:55:29.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:55:29.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:55:29.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:55:29.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:29.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:55:29.882 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:55:29.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:29.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:30.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:30.357 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:30.355+0000 7f4cf100d640 -1 Falling back to public interface 2024-09-17T16:55:31.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:31.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:31.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:55:31.309 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:55:31.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:31.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:31.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:55:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:55:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:33.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:33.386 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:33.384+0000 7f4cf100d640 -1 osd.1 61 log_to_monitors true 2024-09-17T16:55:34.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:34.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:34.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:55:34.139 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:55:34.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:34.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:34.670 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 67 up_thru 67 down_at 63 last_clean_interval [45,61) [v2:127.0.0.1:6810/3844624999,v1:127.0.0.1:6811/3844624999] [v2:127.0.0.1:6812/3844624999,v1:127.0.0.1:6813/3844624999] exists,up 94215679-4d44-47a5-8a6a-ab5394377388 2024-09-17T16:55:34.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:55:34.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:55:34.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:55:34.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:55:34.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:55:34.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:55:34.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:55:34.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:55:34.672 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:55:34.672 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:55:34.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:55:34.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:55:34.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:55:34.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:55:34.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:55:34.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:55:34.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:55:34.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:55:34.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:55:35.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:55:35.149 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:55:35.149 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:55:35.149 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:55:35.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:55:35.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:35.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:55:35.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803820 2024-09-17T16:55:35.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803820 2024-09-17T16:55:35.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803820' 2024-09-17T16:55:35.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:35.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:55:35.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808835 2024-09-17T16:55:35.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808835 2024-09-17T16:55:35.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803820 1-287762808835' 2024-09-17T16:55:35.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:35.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:55:35.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=240518168584 2024-09-17T16:55:35.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 240518168584 2024-09-17T16:55:35.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803820 1-287762808835 2-240518168584' 2024-09-17T16:55:35.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:35.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:55:35.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018901 2024-09-17T16:55:35.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018901 2024-09-17T16:55:35.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803820 1-287762808835 2-240518168584 3-128849018901' 2024-09-17T16:55:35.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:35.698 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803820 2024-09-17T16:55:35.698 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:35.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:55:35.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803820 2024-09-17T16:55:35.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:35.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803820 2024-09-17T16:55:35.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803820' 2024-09-17T16:55:35.700 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803820 2024-09-17T16:55:35.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:55:36.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803820 -lt 25769803820 2024-09-17T16:55:36.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:36.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-287762808835 2024-09-17T16:55:36.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:36.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:55:36.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-287762808835 2024-09-17T16:55:36.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:36.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808835 2024-09-17T16:55:36.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 287762808835' 2024-09-17T16:55:36.123 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 287762808835 2024-09-17T16:55:36.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:55:36.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 287762808835 2024-09-17T16:55:36.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:55:37.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:55:37.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:55:37.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808835 -lt 287762808835 2024-09-17T16:55:37.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:37.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-240518168584 2024-09-17T16:55:37.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:37.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:55:37.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-240518168584 2024-09-17T16:55:37.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:37.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=240518168584 2024-09-17T16:55:37.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 240518168584' 2024-09-17T16:55:37.944 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 240518168584 2024-09-17T16:55:37.944 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:55:38.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 240518168584 -lt 240518168584 2024-09-17T16:55:38.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:38.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018901 2024-09-17T16:55:38.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:38.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:55:38.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018901 2024-09-17T16:55:38.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:38.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018901 2024-09-17T16:55:38.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018901' 2024-09-17T16:55:38.371 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 128849018901 2024-09-17T16:55:38.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:55:38.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018901 -lt 128849018901 2024-09-17T16:55:38.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:55:38.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:55:38.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:55:39.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:55:39.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:55:39.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:55:39.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:55:39.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:55:39.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:55:39.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:55:39.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:55:39.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:55:39.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:55:39.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:55:39.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:55:40.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:55:40.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:55:40.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:55:40.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-17T16:55:40.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-17T16:55:40.962 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-17T16:55:40.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87940-1-0 2024-09-17T16:55:40.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:55:40.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:55:40.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87940-1-0 2024-09-17T16:55:40.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:55:40.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:55:40.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-87940-1-0 td/test-erasure-eio/COPY 2024-09-17T16:55:41.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:55:41.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:55:41.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 1 + 1 2024-09-17T16:55:41.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=2 2024-09-17T16:55:41.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-87940-1-0 td/test-erasure-eio 2 0 set 2024-09-17T16:55:41.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-87940-1-0 2024-09-17T16:55:41.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-17T16:55:41.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-17T16:55:41.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-17T16:55:41.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=2 2024-09-17T16:55:41.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-17T16:55:41.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2024-09-17T16:55:41.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-17T16:55:41.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-17T16:55:41.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-17T16:55:41.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-87940-1-0 2024-09-17T16:55:41.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:55:41.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-87940-1-0 2024-09-17T16:55:41.104 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-87940-1-0 2024-09-17T16:55:41.104 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:55:41.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:55:41.567 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:55:41.567 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:55:41.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T16:55:41.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-17T16:55:41.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-17T16:55:41.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=2 2024-09-17T16:55:41.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-17T16:55:42.250 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-17T16:55:42.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-17T16:55:42.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2024-09-17T16:55:42.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-17T16:55:42.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-87940-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:55:42.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:55:42.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:55:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-17T16:55:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:55:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-87940-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:55:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:55:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:55:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-17T16:55:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:55:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-17T16:55:42.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:55:42.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:55:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:55:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:55:42.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:55:43.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:55:43.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-87940-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:55:43.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:55:43.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:55:43.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-17T16:55:43.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:55:43.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-17T16:55:43.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-87940-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:55:45.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-17T16:55:45.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:55:45.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:55:45.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T16:55:45.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:55:45.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:55:45.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:55:45.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:55:45.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:55:45.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:55:45.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:55:45.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:55:45.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:55:45.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:55:45.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:55:45.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:55:45.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:55:45.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:55:45.510 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:55:45.510 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:55:45.510 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:55:45.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:55:45.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:55:45.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:55:45.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:55:45.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:55:45.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:55:45.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:55:45.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:55:45.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:55:45.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:55:45.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:55:45.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:55:45.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T16:55:45.512 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-09-17T16:55:45.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:55:45.512 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-17T16:55:45.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T16:55:45.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:55:45.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:55:45.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:55:45.539 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:45.537+0000 7fc6f0ba2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:45.568 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:45.566+0000 7fc6f0ba2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:45.570 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:45.567+0000 7fc6f0ba2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T16:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:45.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:55:46.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:46.688 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:46.686+0000 7fc6f0ba2640 -1 Falling back to public interface 2024-09-17T16:55:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:55:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:55:47.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:47.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:55:47.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:55:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:55:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:55:49.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:49.809 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:55:49.807+0000 7fc6f0ba2640 -1 osd.2 72 log_to_monitors true 2024-09-17T16:55:50.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:50.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:50.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:55:50.197 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:55:50.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:50.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:55:50.786 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 78 up_thru 63 down_at 74 last_clean_interval [56,72) [v2:127.0.0.1:6818/1629032448,v1:127.0.0.1:6819/1629032448] [v2:127.0.0.1:6820/1629032448,v1:127.0.0.1:6821/1629032448] exists,up 0be25c09-9bd9-40c5-9fa5-fc87d2b8d363 2024-09-17T16:55:50.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:55:50.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:55:50.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:55:50.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:55:50.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:55:50.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:55:50.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:55:50.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:55:50.787 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:55:50.787 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:55:50.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:55:50.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:55:50.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:55:50.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:55:50.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:55:50.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:55:50.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:55:50.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:55:50.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:55:51.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:55:51.481 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:55:51.481 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:55:51.481 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:55:51.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:55:51.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:51.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:55:51.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803825 2024-09-17T16:55:51.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803825 2024-09-17T16:55:51.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803825' 2024-09-17T16:55:51.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:51.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:55:51.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808840 2024-09-17T16:55:51.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808840 2024-09-17T16:55:51.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803825 1-287762808840' 2024-09-17T16:55:51.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:51.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:55:51.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449091 2024-09-17T16:55:51.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449091 2024-09-17T16:55:51.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803825 1-287762808840 2-335007449091' 2024-09-17T16:55:51.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:55:51.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:55:52.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018906 2024-09-17T16:55:52.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018906 2024-09-17T16:55:52.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803825 1-287762808840 2-335007449091 3-128849018906' 2024-09-17T16:55:52.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:52.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803825 2024-09-17T16:55:52.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:52.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:55:52.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803825 2024-09-17T16:55:52.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:52.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803825 2024-09-17T16:55:52.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803825' 2024-09-17T16:55:52.078 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803825 2024-09-17T16:55:52.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:55:52.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803823 -lt 25769803825 2024-09-17T16:55:52.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:55:53.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:55:53.506 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:55:53.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803825 -lt 25769803825 2024-09-17T16:55:53.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:53.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-287762808840 2024-09-17T16:55:53.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:53.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:55:53.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-287762808840 2024-09-17T16:55:53.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:53.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808840 2024-09-17T16:55:53.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 287762808840' 2024-09-17T16:55:53.935 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 287762808840 2024-09-17T16:55:53.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:55:54.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808840 -lt 287762808840 2024-09-17T16:55:54.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:54.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-335007449091 2024-09-17T16:55:54.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:54.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:55:54.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-335007449091 2024-09-17T16:55:54.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:54.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449091 2024-09-17T16:55:54.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 335007449091' 2024-09-17T16:55:54.352 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 335007449091 2024-09-17T16:55:54.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:55:54.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449091 -lt 335007449091 2024-09-17T16:55:54.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:55:54.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018906 2024-09-17T16:55:54.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:55:54.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:55:54.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018906 2024-09-17T16:55:54.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:55:54.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018906 2024-09-17T16:55:54.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018906' 2024-09-17T16:55:54.761 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 128849018906 2024-09-17T16:55:54.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:55:55.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018906 -lt 128849018906 2024-09-17T16:55:55.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:55:55.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:55:55.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:55:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:55:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:55:55.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:55:55.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:55:55.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:55:55.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:55:55.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:55:55.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:55:56.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:55:56.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:55:56.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:55:56.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:55:56.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:55:56.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:55:56.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:55:56.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-17T16:55:56.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-17T16:55:57.326 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-17T16:55:57.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 1 -eq 1 -a 2 -eq 2 ']' 2024-09-17T16:55:57.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87940-1-0 2024-09-17T16:55:57.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:55:57.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:55:57.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87940-1-0 2024-09-17T16:55:57.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:55:57.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:55:57.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-87940-1-0 td/test-erasure-eio/COPY 2024-09-17T16:55:57.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:55:57.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:55:57.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-17T16:55:57.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:345: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 256 add 2024-09-17T16:55:57.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-17T16:55:57.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-17T16:55:57.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=1 2024-09-17T16:55:57.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-17T16:55:57.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=256 2024-09-17T16:55:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-17T16:55:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=add 2024-09-17T16:55:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-17T16:55:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-87940-1-256 2024-09-17T16:55:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-87940-1-256 2024-09-17T16:55:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:55:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:55:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-87940-1-256 2024-09-17T16:55:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:55:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:55:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:55:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:55:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:55:57.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:55:57.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:55:57.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:55:57.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-87940-1-256 td/test-erasure-eio/ORIGINAL 2024-09-17T16:55:57.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-87940-1-256 td/test-erasure-eio 1 256 add 2024-09-17T16:55:57.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-87940-1-256 2024-09-17T16:55:57.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-17T16:55:57.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-17T16:55:57.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-17T16:55:57.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-17T16:55:57.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-17T16:55:57.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2024-09-17T16:55:57.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-17T16:55:57.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2024-09-17T16:55:57.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-17T16:55:57.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-87940-1-256 2024-09-17T16:55:57.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:55:57.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-87940-1-256 2024-09-17T16:55:57.675 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-87940-1-256 2024-09-17T16:55:57.675 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:55:58.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:55:58.178 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:55:58.178 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:55:58.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T16:55:58.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-17T16:55:58.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-17T16:55:58.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-17T16:55:58.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-17T16:55:58.901 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-17T16:55:58.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2024-09-17T16:55:58.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87940-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:55:58.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:55:58.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:55:58.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T16:55:58.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:55:58.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87940-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:55:58.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:55:58.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:55:58.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T16:55:58.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:55:58.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-17T16:55:58.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:55:58.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:55:58.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:55:58.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:55:58.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:55:59.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:55:59.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87940-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:55:59.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:55:59.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:55:59.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T16:55:59.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:55:59.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-17T16:55:59.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87940-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:56:00.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-17T16:56:00.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:56:00.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:56:00.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T16:56:00.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:56:00.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:56:00.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:56:00.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:56:00.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:56:00.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:56:00.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:56:00.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:56:00.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:56:00.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:56:00.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:56:00.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:56:00.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:56:00.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:56:00.114 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:56:00.114 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:56:00.114 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:56:00.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:56:00.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:56:00.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:56:00.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:56:00.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:56:00.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:56:00.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:56:00.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:56:00.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:56:00.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:56:00.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:56:00.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:56:00.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T16:56:00.115 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-17T16:56:00.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:56:00.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-17T16:56:00.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T16:56:00.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:56:00.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:56:00.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:56:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:00.141+0000 7f35cbd17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:00.204 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:00.202+0000 7f35cbd17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:00.205 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:00.203+0000 7f35cbd17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:00.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T16:56:00.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:56:00.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:56:00.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:56:00.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:56:00.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:00.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:56:00.528 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:56:00.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:00.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:56:00.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:01.784 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:01.782+0000 7f35cbd17640 -1 Falling back to public interface 2024-09-17T16:56:01.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:01.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:01.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:56:01.946 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:56:01.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:01.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:56:02.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:03.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:03.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:03.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:56:03.342 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:56:03.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:03.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:56:03.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:04.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:04.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:04.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:56:04.790 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:56:04.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:04.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:56:04.876 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:04.874+0000 7f35cbd17640 -1 osd.1 82 log_to_monitors true 2024-09-17T16:56:05.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:06.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:06.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:06.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:56:06.255 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-09-17T16:56:06.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:06.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:56:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 89 up_thru 89 down_at 85 last_clean_interval [67,82) [v2:127.0.0.1:6810/317391580,v1:127.0.0.1:6811/317391580] [v2:127.0.0.1:6812/317391580,v1:127.0.0.1:6813/317391580] exists,up 94215679-4d44-47a5-8a6a-ab5394377388 2024-09-17T16:56:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:56:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:56:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:56:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:56:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:56:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:56:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:56:06.926 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:56:06.926 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:56:06.927 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:56:06.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:56:06.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:56:06.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:56:07.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:56:07.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:56:07.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:56:07.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:56:07.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:56:07.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:56:07.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:56:07.457 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:56:07.457 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:56:07.457 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:56:07.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:56:07.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:07.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:56:07.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803830 2024-09-17T16:56:07.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803830 2024-09-17T16:56:07.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803830' 2024-09-17T16:56:07.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:07.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:56:07.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=382252089347 2024-09-17T16:56:07.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 382252089347 2024-09-17T16:56:07.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803830 1-382252089347' 2024-09-17T16:56:07.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:07.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:56:07.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449096 2024-09-17T16:56:07.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449096 2024-09-17T16:56:07.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803830 1-382252089347 2-335007449096' 2024-09-17T16:56:07.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:07.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:56:07.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018911 2024-09-17T16:56:07.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018911 2024-09-17T16:56:07.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803830 1-382252089347 2-335007449096 3-128849018911' 2024-09-17T16:56:07.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:07.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803830 2024-09-17T16:56:07.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:07.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:56:07.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803830 2024-09-17T16:56:07.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:07.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803830 2024-09-17T16:56:07.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803830' 2024-09-17T16:56:07.909 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803830 2024-09-17T16:56:07.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:56:08.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803828 -lt 25769803830 2024-09-17T16:56:08.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:56:09.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:56:09.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:56:09.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803830 -lt 25769803830 2024-09-17T16:56:09.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:09.857 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-382252089347 2024-09-17T16:56:09.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:09.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:56:09.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-382252089347 2024-09-17T16:56:09.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:09.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=382252089347 2024-09-17T16:56:09.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 382252089347' 2024-09-17T16:56:09.858 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 382252089347 2024-09-17T16:56:09.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:56:10.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 382252089347 -lt 382252089347 2024-09-17T16:56:10.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:10.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-335007449096 2024-09-17T16:56:10.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:10.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:56:10.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-335007449096 2024-09-17T16:56:10.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:10.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449096 2024-09-17T16:56:10.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 335007449096' 2024-09-17T16:56:10.244 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 335007449096 2024-09-17T16:56:10.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:56:10.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449096 -lt 335007449096 2024-09-17T16:56:10.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:10.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018911 2024-09-17T16:56:10.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:10.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:56:10.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018911 2024-09-17T16:56:10.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:10.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018911 2024-09-17T16:56:10.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018911' 2024-09-17T16:56:10.633 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 128849018911 2024-09-17T16:56:10.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:56:11.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018912 -lt 128849018911 2024-09-17T16:56:11.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:56:11.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:56:11.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:56:11.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:56:11.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:56:11.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:56:11.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:56:11.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:56:11.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:56:11.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:56:11.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:56:11.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:56:11.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:56:11.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:56:11.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:56:12.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:56:12.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:56:12.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:56:12.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-17T16:56:12.489 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-17T16:56:12.489 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-17T16:56:12.489 INFO:tasks.workunit.client.0.smithi162.stderr:256 bytes copied, 3.315e-05 s, 7.7 MB/s 2024-09-17T16:56:12.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87940-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:56:12.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:56:12.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:56:12.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T16:56:12.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:56:12.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87940-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:56:12.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:56:12.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:56:12.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T16:56:12.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:56:12.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-17T16:56:12.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:56:12.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:56:12.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:56:12.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:56:12.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:56:12.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:56:12.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87940-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:56:12.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:56:12.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:56:12.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T16:56:12.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:56:12.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-17T16:56:12.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87940-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:56:15.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-17T16:56:15.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:56:15.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:56:15.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T16:56:15.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:56:15.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:56:15.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:56:15.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:56:15.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:56:15.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:56:15.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:56:15.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:56:15.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:56:15.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:56:15.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:56:15.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:56:15.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:56:15.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:56:15.272 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:56:15.272 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:56:15.273 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:56:15.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:56:15.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:56:15.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:56:15.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:56:15.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:56:15.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:56:15.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:56:15.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:56:15.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:56:15.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:56:15.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:56:15.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:56:15.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T16:56:15.275 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-17T16:56:15.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:56:15.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-17T16:56:15.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T16:56:15.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:56:15.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:56:15.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:56:15.303 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:15.300+0000 7fbef200c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:15.305 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:15.303+0000 7fbef200c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:15.306 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:15.304+0000 7fbef200c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:15.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T16:56:15.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:56:15.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:56:15.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:56:15.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:56:15.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:15.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:56:15.737 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:56:15.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:15.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:56:16.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:16.427 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:16.425+0000 7fbef200c640 -1 Falling back to public interface 2024-09-17T16:56:17.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:17.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:17.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:56:17.163 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:56:17.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:17.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:56:17.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:18.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:18.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:18.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:56:18.573 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:56:18.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:18.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:56:18.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:19.097 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:19.095+0000 7fbef200c640 -1 osd.1 91 log_to_monitors true 2024-09-17T16:56:19.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:19.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:19.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:56:19.978 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:56:19.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:19.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:56:20.474 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 96 up_thru 96 down_at 92 last_clean_interval [89,91) [v2:127.0.0.1:6810/3701553343,v1:127.0.0.1:6811/3701553343] [v2:127.0.0.1:6812/3701553343,v1:127.0.0.1:6813/3701553343] exists,up 94215679-4d44-47a5-8a6a-ab5394377388 2024-09-17T16:56:20.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:56:20.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:56:20.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:56:20.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:56:20.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:56:20.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:56:20.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:56:20.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:56:20.475 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:56:20.476 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:56:20.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:56:20.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:56:20.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:56:20.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:56:20.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:56:20.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:56:20.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:56:20.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:56:20.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:56:21.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:56:21.105 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:56:21.105 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:56:21.105 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:56:21.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:56:21.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:21.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:56:21.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803835 2024-09-17T16:56:21.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803835 2024-09-17T16:56:21.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835' 2024-09-17T16:56:21.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:21.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:56:21.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860419 2024-09-17T16:56:21.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860419 2024-09-17T16:56:21.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-412316860419' 2024-09-17T16:56:21.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:21.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:56:21.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449101 2024-09-17T16:56:21.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449101 2024-09-17T16:56:21.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-412316860419 2-335007449101' 2024-09-17T16:56:21.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:21.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:56:21.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018916 2024-09-17T16:56:21.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018916 2024-09-17T16:56:21.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-412316860419 2-335007449101 3-128849018916' 2024-09-17T16:56:21.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:21.581 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803835 2024-09-17T16:56:21.581 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:21.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:56:21.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803835 2024-09-17T16:56:21.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:21.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803835 2024-09-17T16:56:21.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803835' 2024-09-17T16:56:21.584 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803835 2024-09-17T16:56:21.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:56:22.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803835 -lt 25769803835 2024-09-17T16:56:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:22.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-412316860419 2024-09-17T16:56:22.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:22.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:56:22.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-412316860419 2024-09-17T16:56:22.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:22.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860419 2024-09-17T16:56:22.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 412316860419' 2024-09-17T16:56:22.018 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 412316860419 2024-09-17T16:56:22.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:56:22.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860419 -lt 412316860419 2024-09-17T16:56:22.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:22.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-335007449101 2024-09-17T16:56:22.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:22.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:56:22.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-335007449101 2024-09-17T16:56:22.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:22.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449101 2024-09-17T16:56:22.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 335007449101' 2024-09-17T16:56:22.444 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 335007449101 2024-09-17T16:56:22.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:56:22.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449099 -lt 335007449101 2024-09-17T16:56:22.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:56:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:56:23.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:56:24.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449101 -lt 335007449101 2024-09-17T16:56:24.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:24.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018916 2024-09-17T16:56:24.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:24.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:56:24.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018916 2024-09-17T16:56:24.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:24.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018916 2024-09-17T16:56:24.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018916' 2024-09-17T16:56:24.265 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 128849018916 2024-09-17T16:56:24.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:56:24.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018916 -lt 128849018916 2024-09-17T16:56:24.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:56:24.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:56:24.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:56:25.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:56:25.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:56:25.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:56:25.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:56:25.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:56:25.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:56:25.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:56:25.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:56:25.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:56:25.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:56:25.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:56:25.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:56:26.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:56:26.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:56:26.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:56:26.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-17T16:56:26.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-17T16:56:26.756 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-17T16:56:26.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87940-1-256 2024-09-17T16:56:26.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:56:26.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:56:26.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87940-1-256 2024-09-17T16:56:26.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:56:26.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:56:26.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-87940-1-256 td/test-erasure-eio/COPY 2024-09-17T16:56:26.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:56:26.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:56:26.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 1 + 1 2024-09-17T16:56:26.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=2 2024-09-17T16:56:26.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-87940-1-256 td/test-erasure-eio 2 256 add 2024-09-17T16:56:26.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-87940-1-256 2024-09-17T16:56:26.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-17T16:56:26.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-17T16:56:26.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-17T16:56:26.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=2 2024-09-17T16:56:26.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-17T16:56:26.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2024-09-17T16:56:26.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-17T16:56:26.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2024-09-17T16:56:26.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-17T16:56:26.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-87940-1-256 2024-09-17T16:56:26.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:56:26.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-87940-1-256 2024-09-17T16:56:26.862 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-87940-1-256 2024-09-17T16:56:26.862 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:56:27.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:56:27.290 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:56:27.290 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:56:27.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T16:56:27.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-17T16:56:27.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-17T16:56:27.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=2 2024-09-17T16:56:27.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-17T16:56:28.012 INFO:tasks.workunit.client.0.smithi162.stderr:noout is set 2024-09-17T16:56:28.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2024-09-17T16:56:28.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-87940-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:56:28.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:56:28.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:56:28.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-17T16:56:28.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:56:28.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-87940-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:56:28.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:56:28.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:56:28.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-17T16:56:28.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:56:28.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-17T16:56:28.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:56:28.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:56:28.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:56:28.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:56:28.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:56:28.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:56:28.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-87940-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:56:28.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:56:28.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:56:28.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-17T16:56:28.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:56:28.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-17T16:56:28.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-87940-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:56:29.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-17T16:56:29.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:56:29.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:56:29.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T16:56:29.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:56:29.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:56:29.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:56:29.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:56:29.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:56:29.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:56:29.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:56:29.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:56:29.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:56:29.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:56:29.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:56:29.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:56:29.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:56:29.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:56:29.036 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:56:29.036 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:56:29.036 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:56:29.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:56:29.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:56:29.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:56:29.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:56:29.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:56:29.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:56:29.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:56:29.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:56:29.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:56:29.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:56:29.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:56:29.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:56:29.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T16:56:29.037 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-09-17T16:56:29.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:56:29.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-17T16:56:29.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T16:56:29.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:56:29.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:56:29.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:56:29.061 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:29.058+0000 7fbd44216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:29.148 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:29.145+0000 7fbd44216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:29.149 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:29.146+0000 7fbd44216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:29.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T16:56:29.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:56:29.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:56:29.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:56:29.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:56:29.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:29.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:56:29.460 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:56:29.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:29.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:29.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:30.016 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:30.013+0000 7fbd44216640 -1 Falling back to public interface 2024-09-17T16:56:30.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:30.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:30.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:56:30.881 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:56:30.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:30.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:31.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:32.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:32.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:32.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:56:32.302 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:56:32.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:32.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:32.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:33.355 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:33.352+0000 7fbd44216640 -1 osd.2 102 log_to_monitors true 2024-09-17T16:56:33.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:33.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:33.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:56:33.690 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:56:33.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:33.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:34.329 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 106 up_thru 92 down_at 103 last_clean_interval [78,102) [v2:127.0.0.1:6818/980863160,v1:127.0.0.1:6819/980863160] [v2:127.0.0.1:6820/980863160,v1:127.0.0.1:6821/980863160] exists,up 0be25c09-9bd9-40c5-9fa5-fc87d2b8d363 2024-09-17T16:56:34.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:56:34.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:56:34.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:56:34.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:56:34.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:56:34.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:56:34.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:56:34.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:56:34.331 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:56:34.331 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:56:34.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:56:34.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:56:34.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:56:34.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:56:34.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:56:34.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:56:34.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:56:34.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:56:34.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:56:34.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:56:34.876 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:56:34.876 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:56:34.876 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:56:34.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:56:34.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:34.877 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:56:35.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803840 2024-09-17T16:56:35.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803840 2024-09-17T16:56:35.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803840' 2024-09-17T16:56:35.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:35.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:56:35.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860424 2024-09-17T16:56:35.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860424 2024-09-17T16:56:35.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803840 1-412316860424' 2024-09-17T16:56:35.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:35.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:56:35.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533379 2024-09-17T16:56:35.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533379 2024-09-17T16:56:35.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803840 1-412316860424 2-455266533379' 2024-09-17T16:56:35.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:35.295 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:56:35.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018921 2024-09-17T16:56:35.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018921 2024-09-17T16:56:35.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803840 1-412316860424 2-455266533379 3-128849018921' 2024-09-17T16:56:35.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:35.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803840 2024-09-17T16:56:35.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:35.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:56:35.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803840 2024-09-17T16:56:35.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:35.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803840 2024-09-17T16:56:35.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803840' 2024-09-17T16:56:35.411 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803840 2024-09-17T16:56:35.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:56:35.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803840 -lt 25769803840 2024-09-17T16:56:35.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:35.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-412316860424 2024-09-17T16:56:35.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:35.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:56:35.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-412316860424 2024-09-17T16:56:35.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:35.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860424 2024-09-17T16:56:35.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 412316860424' 2024-09-17T16:56:35.845 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 412316860424 2024-09-17T16:56:35.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:56:36.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860424 -lt 412316860424 2024-09-17T16:56:36.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:36.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-455266533379 2024-09-17T16:56:36.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:36.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:56:36.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-455266533379 2024-09-17T16:56:36.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:36.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533379 2024-09-17T16:56:36.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 455266533379' 2024-09-17T16:56:36.232 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 455266533379 2024-09-17T16:56:36.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:56:36.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533379 -lt 455266533379 2024-09-17T16:56:36.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:36.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018921 2024-09-17T16:56:36.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:36.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:56:36.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018921 2024-09-17T16:56:36.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:36.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018921 2024-09-17T16:56:36.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018921' 2024-09-17T16:56:36.635 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 128849018921 2024-09-17T16:56:36.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:56:37.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018919 -lt 128849018921 2024-09-17T16:56:37.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:56:38.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:56:38.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:56:38.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018921 -lt 128849018921 2024-09-17T16:56:38.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:56:38.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:56:38.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:56:38.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:56:38.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:56:38.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:56:38.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:56:38.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:56:38.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:56:38.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:56:38.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:56:39.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:56:39.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:56:39.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:56:39.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:56:39.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:56:39.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:56:39.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:56:39.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-17T16:56:39.987 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-09-17T16:56:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-09-17T16:56:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:256 bytes copied, 3.4116e-05 s, 7.5 MB/s 2024-09-17T16:56:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-87940-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:56:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T16:56:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T16:56:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-17T16:56:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T16:56:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-87940-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:56:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T16:56:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T16:56:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-17T16:56:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T16:56:39.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-17T16:56:39.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:56:39.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:56:39.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:56:39.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:56:39.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:56:41.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:56:41.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-87940-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:56:41.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T16:56:41.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T16:56:41.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-17T16:56:41.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T16:56:41.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-17T16:56:41.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-87940-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-17T16:56:43.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-17T16:56:43.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T16:56:43.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T16:56:43.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T16:56:43.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T16:56:43.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:56:43.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:56:43.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:56:43.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:56:43.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:56:43.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:56:43.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:56:43.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T16:56:43.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T16:56:43.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:56:43.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T16:56:43.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:56:43.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:56:43.330 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:56:43.330 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:56:43.330 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:56:43.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:56:43.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:56:43.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:56:43.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:56:43.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:56:43.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:56:43.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:56:43.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:56:43.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:56:43.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T16:56:43.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T16:56:43.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:56:43.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T16:56:43.331 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-09-17T16:56:43.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:56:43.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-17T16:56:43.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T16:56:43.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T16:56:43.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T16:56:43.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T16:56:43.358 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:43.356+0000 7fc6fb80b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:43.361 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:43.359+0000 7fc6fb80b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:43.362 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:43.360+0000 7fc6fb80b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:43.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T16:56:43.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:56:43.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:56:43.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:56:43.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:56:43.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:43.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:56:43.793 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T16:56:43.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:43.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:44.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:44.943 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:44.941+0000 7fc6fb80b640 -1 Falling back to public interface 2024-09-17T16:56:45.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:45.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:45.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:56:45.203 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:56:45.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:45.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:45.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:46.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:46.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:46.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:56:46.600 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:56:46.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:46.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:46.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:47.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:56:47.839+0000 7fc6fb80b640 -1 osd.2 108 log_to_monitors true 2024-09-17T16:56:47.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:47.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:47.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:56:47.993 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T16:56:47.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:47.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:48.804 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 112 up_thru 92 down_at 109 last_clean_interval [106,108) [v2:127.0.0.1:6818/2434924167,v1:127.0.0.1:6819/2434924167] [v2:127.0.0.1:6820/2434924167,v1:127.0.0.1:6821/2434924167] exists,up 0be25c09-9bd9-40c5-9fa5-fc87d2b8d363 2024-09-17T16:56:48.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:56:48.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:56:48.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:56:48.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T16:56:48.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:56:48.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:56:48.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:56:48.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:56:48.806 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:56:48.806 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:56:48.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:56:48.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:56:48.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:56:48.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:56:48.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:56:48.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:56:48.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:56:48.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:56:48.896 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:56:49.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:56:49.394 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:56:49.394 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:56:49.394 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:56:49.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:56:49.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:49.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:56:49.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803844 2024-09-17T16:56:49.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803844 2024-09-17T16:56:49.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803844' 2024-09-17T16:56:49.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:49.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:56:49.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860429 2024-09-17T16:56:49.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860429 2024-09-17T16:56:49.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803844 1-412316860429' 2024-09-17T16:56:49.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:49.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:56:49.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=481036337155 2024-09-17T16:56:49.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 481036337155 2024-09-17T16:56:49.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803844 1-412316860429 2-481036337155' 2024-09-17T16:56:49.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:56:49.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:56:50.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018926 2024-09-17T16:56:50.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018926 2024-09-17T16:56:50.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803844 1-412316860429 2-481036337155 3-128849018926' 2024-09-17T16:56:50.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:50.034 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803844 2024-09-17T16:56:50.034 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:50.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:56:50.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803844 2024-09-17T16:56:50.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:50.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803844 2024-09-17T16:56:50.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803844' 2024-09-17T16:56:50.037 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803844 2024-09-17T16:56:50.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:56:50.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803842 -lt 25769803844 2024-09-17T16:56:50.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:56:51.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:56:51.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:56:51.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803845 -lt 25769803844 2024-09-17T16:56:51.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:51.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-412316860429 2024-09-17T16:56:51.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:51.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:56:51.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-412316860429 2024-09-17T16:56:51.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:51.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860429 2024-09-17T16:56:51.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 412316860429' 2024-09-17T16:56:51.852 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 412316860429 2024-09-17T16:56:51.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:56:52.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860429 -lt 412316860429 2024-09-17T16:56:52.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:52.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-481036337155 2024-09-17T16:56:52.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:52.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:56:52.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-481036337155 2024-09-17T16:56:52.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:52.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=481036337155 2024-09-17T16:56:52.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 481036337155' 2024-09-17T16:56:52.272 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 481036337155 2024-09-17T16:56:52.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:56:52.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 481036337155 -lt 481036337155 2024-09-17T16:56:52.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:56:52.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018926 2024-09-17T16:56:52.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:56:52.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:56:52.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018926 2024-09-17T16:56:52.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:56:52.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018926 2024-09-17T16:56:52.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018926' 2024-09-17T16:56:52.732 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 128849018926 2024-09-17T16:56:52.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:56:53.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018926 -lt 128849018926 2024-09-17T16:56:53.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:56:53.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:56:53.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:56:53.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:56:53.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:56:53.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:56:53.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:56:53.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:56:53.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:56:53.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:56:53.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:56:54.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:56:54.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:56:54.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:56:54.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:56:54.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:56:54.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:56:54.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:56:54.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-17T16:56:54.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-17T16:56:55.230 INFO:tasks.workunit.client.0.smithi162.stderr:noout is unset 2024-09-17T16:56:55.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 1 -eq 1 -a 2 -eq 2 ']' 2024-09-17T16:56:55.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87940-1-256 2024-09-17T16:56:55.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:56:55.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:56:55.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87940-1-256 2024-09-17T16:56:55.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:56:55.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:56:55.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-87940-1-256 td/test-erasure-eio/COPY 2024-09-17T16:56:55.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:56:55.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:56:55.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-17T16:56:55.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:346: TEST_rados_get_bad_size_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-17T16:56:55.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:56:55.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-17T16:56:55.890 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-17T16:56:55.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-17T16:56:56.471 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-17T16:56:56.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-17T16:56:56.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:56:56.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:56:56.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:56:56.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:56:56.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:56:56.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:56:56.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:56:56.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:56:56.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:56:56.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:56:56.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:56:56.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:56:56.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:56:56.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:56:56.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:56:56.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:56:56.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:56:56.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:56:56.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:56:56.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:56:56.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:56:56.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:56:56.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:56:56.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:56:56.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:56:56.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:56:56.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:56:56.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:56:56.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-17T16:56:56.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-17T16:56:56.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-17T16:56:56.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-17T16:56:56.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:56:56.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:56:56.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:56:56.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:56:56.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:56:56.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:56:56.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:56:56.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:56:56.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:56:56.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:56:56.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:56:56.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:56:56.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:56:56.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:56:56.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:56:56.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:56:56.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:56:56.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:56:56.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:56:56.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:56:56.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:56:56.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:56:56.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:56:56.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:56:56.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:56:56.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:56:56.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:56:56.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:56:56.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-17T16:56:56.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:56:56.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:56:56.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:56:56.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.87940 2024-09-17T16:56:56.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:56:56.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T16:56:56.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:56:56.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-17T16:56:56.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-17T16:56:56.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-17T16:56:56.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:56:56.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:56:56.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:56:56.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-17T16:56:56.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-17T16:56:57.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:56:57.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:56:57.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:56:57.096 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:56:57.096 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:56:57.097 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:56:57.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:56:57.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T16:56:57.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:56:57.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:56:57.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:56:57.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:56:57.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:56:57.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:56:57.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:56:57.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:56:57.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:56:57.300 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:56:57.301 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:56:57.301 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:56:57.301 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:56:57.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:56:57.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get fsid 2024-09-17T16:56:57.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:56:57.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:56:57.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:56:57.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:56:57.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:56:57.382 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:56:57.382 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:56:57.382 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:56:57.383 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:56:57.383 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:56:57.383 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:56:57.383 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:56:57.383 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:56:57.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get mon_host 2024-09-17T16:56:57.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-17T16:56:57.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-17T16:56:57.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:56:57.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:56:57.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:56:57.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-17T16:56:57.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:56:58.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:56:58.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:56:58.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:56:58.310 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:56:58.310 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:56:58.310 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:56:58.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:56:58.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:56:58.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:56:58.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-17T16:56:58.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T16:56:58.637 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T16:56:58.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:56:59.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-17T16:56:59.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:56:59.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:56:59.649 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:56:59.650 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:56:59.650 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:56:59.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:56:59.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-17T16:56:59.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-mon.a.asok log flush 2024-09-17T16:56:59.721 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:56:59.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-17T16:56:59.730 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:56:57.128+0000 7f6c1711ec80 0 load: jerasure load: lrc 2024-09-17T16:56:59.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_0 td/test-erasure-eio 2024-09-17T16:56:59.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:275: TEST_rados_get_subread_eio_shard_0: local dir=td/test-erasure-eio 2024-09-17T16:56:59.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:276: TEST_rados_get_subread_eio_shard_0: setup_osds 4 2024-09-17T16:56:59.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-17T16:56:59.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-17T16:56:59.731 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-17T16:56:59.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-17T16:56:59.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:56:59.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-17T16:56:59.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:56:59.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:56:59.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:56:59.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:56:59.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-17T16:56:59.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:56:59.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:56:59.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:56:59.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:56:59.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-17T16:56:59.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-17T16:56:59.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:56:59.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:56:59.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:56:59.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:56:59.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:56:59.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:56:59.738 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:56:59.738 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:56:59.738 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:56:59.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:56:59.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:56:59.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:56:59.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:56:59.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:56:59.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:56:59.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:56:59.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:56:59.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:56:59.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:56:59.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:56:59.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:56:59.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:56:59.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-17T16:56:59.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:56:59.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3620e3d0-0eae-4c1c-9755-3e22b48366ff 2024-09-17T16:56:59.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3620e3d0-0eae-4c1c-9755-3e22b48366ff' 2024-09-17T16:56:59.741 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 3620e3d0-0eae-4c1c-9755-3e22b48366ff 2024-09-17T16:56:59.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:56:59.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDbtOlmckLrLBAAbNPwA9nIpC0f4fUZVRAZsw== 2024-09-17T16:56:59.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDbtOlmckLrLBAAbNPwA9nIpC0f4fUZVRAZsw=="}' 2024-09-17T16:56:59.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3620e3d0-0eae-4c1c-9755-3e22b48366ff -i td/test-erasure-eio/0/new.json 2024-09-17T16:57:00.021 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:57:00.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-17T16:57:00.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbtOlmckLrLBAAbNPwA9nIpC0f4fUZVRAZsw== --osd-uuid 3620e3d0-0eae-4c1c-9755-3e22b48366ff 2024-09-17T16:57:00.061 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:00.059+0000 7f256fa13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:00.090 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:00.088+0000 7f256fa13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:00.091 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:00.089+0000 7f256fa13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:00.092 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:00.090+0000 7f256fa13640 -1 bdev(0x5637b1c8fc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:57:00.092 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:00.090+0000 7f256fa13640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-17T16:57:06.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-17T16:57:06.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:57:06.047 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:57:06.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:57:06.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:57:06.585 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:57:06.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:57:06.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:57:06.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:57:06.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:57:06.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:57:06.613 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:06.611+0000 7fd00e123640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:06.629 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:06.627+0000 7fd00e123640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:06.630 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:06.629+0000 7fd00e123640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:07.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:57:07.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:57:07.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:57:07.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:57:07.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:57:07.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:07.046 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:57:07.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:57:07.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:07.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:07.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:07.980 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:07.978+0000 7fd00e123640 -1 Falling back to public interface 2024-09-17T16:57:08.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:08.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:08.437 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:57:08.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:57:08.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:08.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:08.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:09.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:09.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:09.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:57:09.864 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:57:09.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:09.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:10.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:10.766 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:10.763+0000 7fd00e123640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:57:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:11.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:11.262 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:57:11.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:57:11.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:11.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:11.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:12.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:12.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:12.834 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:57:12.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:57:12.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:12.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:13.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:14.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:14.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:14.264 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:57:14.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:57:14.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:14.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:14.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:15.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:15.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:15.687 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:57:15.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:57:15.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:15.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:16.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:17.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:17.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:17.122 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:57:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:57:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:17.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:17.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:18.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:18.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:18.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:57:18.653 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:57:18.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:18.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:19.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:20.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:20.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:20.076 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:57:20.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:57:20.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:20.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:20.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:21.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:21.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:21.558 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:57:21.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:57:21.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:21.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:21.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:22.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:22.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:22.972 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:57:22.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:57:22.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:22.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:23.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:24.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:24.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:24.842 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:57:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:57:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:25.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:26.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:26.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:26.461 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:57:26.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:57:26.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:26.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:26.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:27.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:27.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:27.852 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:57:27.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:57:27.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:27.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:28.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:29.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:29.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:29.500 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:57:29.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:57:29.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:29.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:30.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:31.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:31.543 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:57:31.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:31.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:57:31.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:31.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:32.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:33.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:33.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:33.381 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:57:33.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:57:33.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:33.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:57:33.817 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1778624999,v1:127.0.0.1:6803/1778624999] [v2:127.0.0.1:6804/1778624999,v1:127.0.0.1:6805/1778624999] exists,up 3620e3d0-0eae-4c1c-9755-3e22b48366ff 2024-09-17T16:57:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:57:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:57:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:57:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:57:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-17T16:57:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:57:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:57:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T16:57:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:57:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T16:57:33.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:57:33.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:57:33.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:57:33.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:57:33.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T16:57:33.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T16:57:33.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:57:33.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:57:33.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:57:33.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:57:33.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:57:33.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:57:33.826 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:57:33.826 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:57:33.827 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:57:33.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:57:33.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:57:33.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:57:33.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:57:33.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:57:33.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:57:33.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:57:33.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:57:33.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:57:33.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:57:33.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:57:33.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:57:33.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:57:33.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T16:57:33.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:57:33.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=beb8bd89-c86c-4b81-add4-a5766594d223 2024-09-17T16:57:33.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 beb8bd89-c86c-4b81-add4-a5766594d223' 2024-09-17T16:57:33.828 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 beb8bd89-c86c-4b81-add4-a5766594d223 2024-09-17T16:57:33.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:57:33.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD9tOlm46XrMRAA+jX/Dc9ArmxnDV5VJjoJXQ== 2024-09-17T16:57:33.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD9tOlm46XrMRAA+jX/Dc9ArmxnDV5VJjoJXQ=="}' 2024-09-17T16:57:33.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new beb8bd89-c86c-4b81-add4-a5766594d223 -i td/test-erasure-eio/1/new.json 2024-09-17T16:57:34.635 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:57:34.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-17T16:57:34.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD9tOlm46XrMRAA+jX/Dc9ArmxnDV5VJjoJXQ== --osd-uuid beb8bd89-c86c-4b81-add4-a5766594d223 2024-09-17T16:57:34.676 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:34.674+0000 7feab7151640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:34.685 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:34.683+0000 7feab7151640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:34.686 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:34.684+0000 7feab7151640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:34.687 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:34.685+0000 7feab7151640 -1 bdev(0x5607cc795c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-17T16:57:34.687 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:34.685+0000 7feab7151640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-17T16:57:40.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-17T16:57:40.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:57:40.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T16:57:40.396 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T16:57:40.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:57:40.975 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T16:57:40.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T16:57:40.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:57:40.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:57:40.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:57:40.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:57:41.003 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:41.001+0000 7ff1d7019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:41.016 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:41.014+0000 7ff1d7019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:41.016+0000 7ff1d7019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:41.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T16:57:41.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:57:41.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:57:41.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:57:41.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:57:41.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:41.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:57:41.431 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:57:41.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:41.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:57:41.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:42.371 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:42.369+0000 7ff1d7019640 -1 Falling back to public interface 2024-09-17T16:57:42.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:42.858 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:57:42.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:42.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:57:42.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:42.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:57:43.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:44.293 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:57:44.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:57:44.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:44.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:57:44.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:45.188 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:57:45.186+0000 7ff1d7019640 -1 osd.1 0 log_to_monitors true 2024-09-17T16:57:45.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:45.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:45.726 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:57:45.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:57:45.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:45.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:57:46.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:47.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:47.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:47.247 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:57:47.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:57:47.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:47.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:57:47.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:48.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:48.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:48.782 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:57:48.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:57:48.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:48.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:57:49.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:50.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:50.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:50.289 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:57:50.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:57:50.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:50.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:57:50.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:51.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:51.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:51.838 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:57:51.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:57:51.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:51.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:57:52.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:53.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:53.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:53.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:57:53.380 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:57:53.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:53.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:57:53.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:54.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:54.779 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:57:54.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:54.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:57:54.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:54.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:57:55.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:56.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:56.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:56.240 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:57:56.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:57:56.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:56.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:57:56.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:57.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:57.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:57.762 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:57:57.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:57:57.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:57.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:57:58.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:59.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:59.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:59.627 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:57:59.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:57:59.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:59.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:58:00.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:01.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:01.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:01.405 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:58:01.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:58:01.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:01.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:58:02.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:03.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:03.207 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:58:03.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:03.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:58:03.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:03.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:58:03.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:04.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:04.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:04.900 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:58:04.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:58:04.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:04.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:58:06.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:07.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:07.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:07.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:58:07.098 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:58:07.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:07.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:58:08.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:09.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:09.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:09.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:58:09.215 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:58:09.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:09.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:58:09.620 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3286359327,v1:127.0.0.1:6811/3286359327] [v2:127.0.0.1:6812/3286359327,v1:127.0.0.1:6813/3286359327] exists,up beb8bd89-c86c-4b81-add4-a5766594d223 2024-09-17T16:58:09.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:58:09.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:58:09.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:58:09.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:58:09.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-17T16:58:09.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:58:09.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:58:09.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T16:58:09.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:58:09.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T16:58:09.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:58:09.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:58:09.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:58:09.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:58:09.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T16:58:09.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T16:58:09.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:58:09.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:58:09.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:58:09.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:58:09.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:58:09.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:58:09.626 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:58:09.626 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:58:09.626 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:58:09.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:58:09.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:58:09.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:58:09.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:58:09.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:58:09.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:58:09.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:58:09.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:58:09.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:58:09.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:58:09.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:58:09.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:58:09.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:58:09.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T16:58:09.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:58:09.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e6dbb9b0-f7e8-4b90-84f8-518a4067a7a5 2024-09-17T16:58:09.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 e6dbb9b0-f7e8-4b90-84f8-518a4067a7a5' 2024-09-17T16:58:09.628 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 e6dbb9b0-f7e8-4b90-84f8-518a4067a7a5 2024-09-17T16:58:09.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:58:09.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAhtelmBcwlJhAAZxyo2ZPM5F9bBDmLoC2LRA== 2024-09-17T16:58:09.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAhtelmBcwlJhAAZxyo2ZPM5F9bBDmLoC2LRA=="}' 2024-09-17T16:58:09.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e6dbb9b0-f7e8-4b90-84f8-518a4067a7a5 -i td/test-erasure-eio/2/new.json 2024-09-17T16:58:10.181 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:58:10.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-17T16:58:10.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhtelmBcwlJhAAZxyo2ZPM5F9bBDmLoC2LRA== --osd-uuid e6dbb9b0-f7e8-4b90-84f8-518a4067a7a5 2024-09-17T16:58:10.222 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:10.220+0000 7f23ae644640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:10.223 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:10.222+0000 7f23ae644640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:10.224 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:10.223+0000 7f23ae644640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:10.225 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:10.223+0000 7f23ae644640 -1 bdev(0x55b4dc7dfc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-17T16:58:10.225 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:10.223+0000 7f23ae644640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-17T16:58:16.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-17T16:58:16.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:58:16.160 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T16:58:16.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T16:58:16.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:58:16.711 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T16:58:16.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T16:58:16.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:58:16.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:58:16.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:58:16.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:58:16.738 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:16.736+0000 7f2617216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:16.752 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:16.750+0000 7f2617216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:16.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:16.751+0000 7f2617216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:17.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T16:58:17.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:58:17.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:58:17.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:58:17.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:58:17.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:17.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:58:17.159 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:58:17.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:17.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:17.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:18.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:18.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:18.585 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:58:18.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:58:18.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:18.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:18.595 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:18.593+0000 7f2617216640 -1 Falling back to public interface 2024-09-17T16:58:19.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:20.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:20.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:20.029 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:58:20.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:58:20.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:20.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:20.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:21.248 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:21.246+0000 7f2617216640 -1 osd.2 0 log_to_monitors true 2024-09-17T16:58:21.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:21.486 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:58:21.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:21.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:58:21.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:21.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:21.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:22.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:22.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:22.970 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:58:22.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:58:22.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:22.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:23.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:24.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:24.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:24.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:58:24.363 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:58:24.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:24.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:24.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:25.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:25.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:25.798 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:58:25.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:58:25.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:25.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:26.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:27.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:27.244 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:58:27.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:27.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:58:27.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:27.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:27.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:28.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:28.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:28.642 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:58:28.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:58:28.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:28.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:29.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:30.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:30.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:30.134 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:58:30.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:58:30.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:30.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:30.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:31.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:31.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:31.605 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:58:31.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:58:31.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:31.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:32.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:33.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:33.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:33.080 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:58:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:58:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:33.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:34.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:34.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:34.684 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:58:34.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:58:34.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:34.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:35.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:36.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:36.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:36.948 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:58:36.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:58:36.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:36.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:37.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:38.767 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:58:38.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:58:38.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:38.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:39.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:40.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:40.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:40.301 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:58:40.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:58:40.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:40.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:40.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:41.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:41.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:41.903 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:58:41.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:58:41.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:41.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:43.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:44.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:44.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:44.192 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:58:44.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:58:44.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:44.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:44.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:45.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:45.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:45.720 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:58:45.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:58:45.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:45.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:58:46.377 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2978661086,v1:127.0.0.1:6819/2978661086] [v2:127.0.0.1:6820/2978661086,v1:127.0.0.1:6821/2978661086] exists,up e6dbb9b0-f7e8-4b90-84f8-518a4067a7a5 2024-09-17T16:58:46.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:58:46.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:58:46.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:58:46.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:58:46.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-17T16:58:46.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:58:46.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:58:46.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T16:58:46.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:58:46.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T16:58:46.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:58:46.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:58:46.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:58:46.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:58:46.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T16:58:46.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T16:58:46.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:58:46.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:58:46.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:58:46.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:58:46.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:58:46.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:58:46.384 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:58:46.385 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:58:46.385 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:58:46.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:58:46.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:58:46.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:58:46.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:58:46.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:58:46.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:58:46.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:58:46.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:58:46.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:58:46.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:58:46.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:58:46.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:58:46.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:58:46.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T16:58:46.386 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:58:46.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7eb6214c-535b-4a07-83c0-cf791a768f9c 2024-09-17T16:58:46.386 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 7eb6214c-535b-4a07-83c0-cf791a768f9c 2024-09-17T16:58:46.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 7eb6214c-535b-4a07-83c0-cf791a768f9c' 2024-09-17T16:58:46.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:58:46.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBGtelm+e6vFxAAOWp1aLPUqk3j6g9uM8vCEg== 2024-09-17T16:58:46.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBGtelm+e6vFxAAOWp1aLPUqk3j6g9uM8vCEg=="}' 2024-09-17T16:58:46.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7eb6214c-535b-4a07-83c0-cf791a768f9c -i td/test-erasure-eio/3/new.json 2024-09-17T16:58:46.978 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:58:46.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-17T16:58:46.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGtelm+e6vFxAAOWp1aLPUqk3j6g9uM8vCEg== --osd-uuid 7eb6214c-535b-4a07-83c0-cf791a768f9c 2024-09-17T16:58:47.020 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:47.017+0000 7f951dd43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:47.022 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:47.019+0000 7f951dd43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:47.023 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:47.020+0000 7f951dd43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:47.023 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:47.020+0000 7f951dd43640 -1 bdev(0x5596d9fc3c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-17T16:58:47.023 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:47.021+0000 7f951dd43640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-17T16:58:52.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-17T16:58:52.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:58:52.741 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-17T16:58:52.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T16:58:52.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:58:53.276 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T16:58:53.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T16:58:53.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:58:53.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:58:53.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:58:53.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:58:53.304 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:53.301+0000 7f9603bbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:53.316 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:53.313+0000 7f9603bbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:53.317 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:53.314+0000 7f9603bbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:53.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T16:58:53.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:58:53.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:58:53.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:58:53.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:58:53.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:53.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:58:53.764 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:58:53.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:53.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:58:54.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:55.168 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:55.165+0000 7f9603bbb640 -1 Falling back to public interface 2024-09-17T16:58:55.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:55.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:55.181 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:58:55.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:58:55.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:55.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:58:55.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:56.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:56.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:56.564 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:58:56.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:58:56.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:56.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:58:56.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:57.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:57.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:57.987 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:58:57.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:58:57.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:57.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:58:58.032 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:58:58.030+0000 7f9603bbb640 -1 osd.3 0 log_to_monitors true 2024-09-17T16:58:58.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:59.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:59.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:59.516 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:58:59.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:58:59.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:59.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:59:00.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:01.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:01.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:01.032 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:59:01.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:59:01.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:01.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:59:01.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:02.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:02.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:02.435 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:59:02.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:59:02.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:02.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:59:02.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:03.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:03.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:03.946 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:59:03.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:59:03.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:03.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:59:04.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:05.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:05.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:05.371 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:59:05.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:59:05.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:05.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:59:05.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:06.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:06.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:06.844 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:59:06.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:59:06.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:06.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:59:07.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:08.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:08.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:08.260 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:59:08.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:59:08.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:08.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:59:08.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:09.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:09.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:09.661 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T16:59:09.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:59:09.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:09.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:59:10.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:11.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:11.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:11.133 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T16:59:11.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:59:11.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:11.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:59:11.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:12.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:12.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:12.809 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T16:59:12.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:59:12.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:12.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:59:14.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:15.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:15.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:15.022 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T16:59:15.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:59:15.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:15.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:59:15.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:16.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:16.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:16.584 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T16:59:16.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:59:16.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:16.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:59:17.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:18.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:18.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:18.678 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T16:59:18.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:59:18.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:18.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:59:19.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:20.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:20.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:20.572 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T16:59:20.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:59:20.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:20.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:59:21.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:22.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:22.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:22.359 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T16:59:22.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T16:59:22.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:22.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:59:22.769 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/745979758,v1:127.0.0.1:6827/745979758] [v2:127.0.0.1:6828/745979758,v1:127.0.0.1:6829/745979758] exists,up 7eb6214c-535b-4a07-83c0-cf791a768f9c 2024-09-17T16:59:22.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:59:22.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:59:22.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:59:22.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-17T16:59:22.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T16:59:22.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T16:59:22.772 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:59:22.772 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:59:22.772 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:59:22.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T16:59:22.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-17T16:59:22.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.0.asok log flush 2024-09-17T16:59:22.850 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:59:22.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-17T16:59:22.861 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:57:07.982+0000 7fd00e123640 0 load: jerasure load: lrc 2024-09-17T16:59:22.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:278: TEST_rados_get_subread_eio_shard_0: local poolname=pool-jerasure 2024-09-17T16:59:22.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:279: TEST_rados_get_subread_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-17T16:59:22.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:59:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-17T16:59:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-17T16:59:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-17T16:59:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-17T16:59:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-17T16:59:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-17T16:59:23.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-17T16:59:23.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-17T16:59:25.024 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-17T16:59:25.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:59:26.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-17T16:59:26.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T16:59:26.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T16:59:26.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T16:59:26.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T16:59:26.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T16:59:26.039 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T16:59:26.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T16:59:26.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T16:59:26.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T16:59:26.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T16:59:26.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T16:59:26.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T16:59:26.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T16:59:26.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T16:59:26.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T16:59:26.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T16:59:26.560 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:59:26.560 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T16:59:26.560 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T16:59:26.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T16:59:26.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:59:26.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T16:59:26.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803806 2024-09-17T16:59:26.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803806 2024-09-17T16:59:26.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806' 2024-09-17T16:59:26.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:59:26.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T16:59:26.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574870 2024-09-17T16:59:26.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574870 2024-09-17T16:59:26.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806 1-55834574870' 2024-09-17T16:59:26.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:59:26.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T16:59:26.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378638 2024-09-17T16:59:26.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378638 2024-09-17T16:59:26.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806 1-55834574870 2-81604378638' 2024-09-17T16:59:26.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T16:59:26.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T16:59:27.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051593 2024-09-17T16:59:27.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051593 2024-09-17T16:59:27.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803806 1-55834574870 2-81604378638 3-124554051593' 2024-09-17T16:59:27.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:59:27.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803806 2024-09-17T16:59:27.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:59:27.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T16:59:27.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803806 2024-09-17T16:59:27.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:59:27.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803806 2024-09-17T16:59:27.017 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803806 2024-09-17T16:59:27.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803806' 2024-09-17T16:59:27.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:59:27.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803804 -lt 25769803806 2024-09-17T16:59:27.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T16:59:28.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T16:59:28.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T16:59:28.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803806 -lt 25769803806 2024-09-17T16:59:28.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:59:28.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574870 2024-09-17T16:59:28.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:59:28.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T16:59:28.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574870 2024-09-17T16:59:28.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:59:28.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574870 2024-09-17T16:59:28.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574870' 2024-09-17T16:59:28.884 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574870 2024-09-17T16:59:28.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T16:59:29.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574870 -lt 55834574870 2024-09-17T16:59:29.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:59:29.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378638 2024-09-17T16:59:29.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:59:29.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T16:59:29.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378638 2024-09-17T16:59:29.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:59:29.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378638 2024-09-17T16:59:29.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378638' 2024-09-17T16:59:29.310 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378638 2024-09-17T16:59:29.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T16:59:29.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378638 -lt 81604378638 2024-09-17T16:59:29.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T16:59:29.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-124554051593 2024-09-17T16:59:29.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T16:59:29.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T16:59:29.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-124554051593 2024-09-17T16:59:29.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T16:59:29.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051593 2024-09-17T16:59:29.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 124554051593' 2024-09-17T16:59:29.736 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 124554051593 2024-09-17T16:59:29.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T16:59:30.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051594 -lt 124554051593 2024-09-17T16:59:30.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T16:59:30.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:59:30.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:59:30.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T16:59:30.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T16:59:30.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T16:59:30.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T16:59:30.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T16:59:30.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T16:59:30.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T16:59:30.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T16:59:31.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T16:59:31.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T16:59:31.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T16:59:31.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T16:59:31.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T16:59:31.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T16:59:31.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T16:59:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:281: TEST_rados_get_subread_eio_shard_0: local shard_id=0 2024-09-17T16:59:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:282: TEST_rados_get_subread_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 2024-09-17T16:59:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-17T16:59:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-17T16:59:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-17T16:59:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-17T16:59:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2024-09-17T16:59:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-17T16:59:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-17T16:59:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-17T16:59:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-87940-0 2024-09-17T16:59:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-87940-0 2024-09-17T16:59:31.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T16:59:31.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T16:59:31.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-87940-0 2024-09-17T16:59:31.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:59:31.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T16:59:31.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:59:31.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T16:59:31.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:59:31.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T16:59:31.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T16:59:31.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T16:59:31.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-87940-0 td/test-erasure-eio/ORIGINAL 2024-09-17T16:59:31.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-87940-0 td/test-erasure-eio 0 2024-09-17T16:59:31.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T16:59:31.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T16:59:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T16:59:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T16:59:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T16:59:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T16:59:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-87940-0 2024-09-17T16:59:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T16:59:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T16:59:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T16:59:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-17T16:59:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T16:59:31.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-87940-0 2024-09-17T16:59:31.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:59:31.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-87940-0 2024-09-17T16:59:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-87940-0 2024-09-17T16:59:31.713 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:59:32.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:59:32.140 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:59:32.140 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:59:32.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T16:59:32.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-17T16:59:32.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T16:59:32.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-17T16:59:32.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T16:59:32.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-17T16:59:32.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T16:59:32.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-17T16:59:32.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T16:59:32.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-17T16:59:32.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T16:59:32.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T16:59:32.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T16:59:32.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-17T16:59:32.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-17T16:59:32.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-17T16:59:32.143 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:59:32.143 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:59:32.143 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:59:32.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.3.asok 2024-09-17T16:59:32.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-17T16:59:32.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T16:59:32.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T16:59:32.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T16:59:32.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-17T16:59:32.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-17T16:59:32.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-17T16:59:32.228 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:59:32.228 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:59:32.228 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:59:32.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.3.asok 2024-09-17T16:59:32.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T16:59:32.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-87940-0 0 2024-09-17T16:59:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-87940-0 2024-09-17T16:59:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:59:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:59:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-87940-0 2024-09-17T16:59:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T16:59:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T16:59:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-87940-0 td/test-erasure-eio/COPY 2024-09-17T16:59:32.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T16:59:32.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T16:59:32.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-17T16:59:32.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 0 + 1 2024-09-17T16:59:32.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: local another_shard_id=1 2024-09-17T16:59:32.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-87940-0 td/test-erasure-eio 1 2024-09-17T16:59:32.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T16:59:32.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T16:59:32.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T16:59:32.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T16:59:32.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T16:59:32.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T16:59:32.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-87940-0 2024-09-17T16:59:32.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T16:59:32.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T16:59:32.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T16:59:32.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-17T16:59:32.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T16:59:32.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-87940-0 2024-09-17T16:59:32.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T16:59:32.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-87940-0 2024-09-17T16:59:32.351 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-87940-0 2024-09-17T16:59:32.351 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T16:59:32.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T16:59:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T16:59:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T16:59:32.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T16:59:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-17T16:59:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T16:59:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-17T16:59:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T16:59:32.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/1/type 2024-09-17T16:59:32.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T16:59:32.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-17T16:59:32.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T16:59:32.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-17T16:59:32.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T16:59:32.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T16:59:32.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T16:59:32.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-17T16:59:32.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T16:59:32.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T16:59:32.779 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:59:32.779 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:59:32.779 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:59:32.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.1.asok 2024-09-17T16:59:32.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-17T16:59:32.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T16:59:32.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T16:59:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T16:59:32.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-17T16:59:32.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T16:59:32.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T16:59:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:59:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:59:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:59:32.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.1.asok 2024-09-17T16:59:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T16:59:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-87940-0 1 2024-09-17T16:59:32.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: '[' 0 -eq 1 -a 1 -eq 2 ']' 2024-09-17T16:59:32.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-87940-0 fail 2024-09-17T16:59:32.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T16:59:32.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T16:59:32.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-87940-0 2024-09-17T16:59:32.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-17T16:59:32.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-17T16:59:32.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-eio-87940-0 td/test-erasure-eio/COPY 2024-09-17T16:59:32.981 INFO:tasks.workunit.client.0.smithi162.stderr:error getting pool-jerasure/obj-eio-87940-0: (5) Input/output error 2024-09-17T16:59:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-17T16:59:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-17T16:59:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:283: TEST_rados_get_subread_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-17T16:59:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T16:59:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-17T16:59:33.487 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-17T16:59:33.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-17T16:59:34.194 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-17T16:59:34.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-17T16:59:34.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:59:34.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:59:34.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:59:34.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:59:34.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:59:34.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:59:34.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:59:34.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:59:34.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:59:34.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:59:34.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:59:34.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:59:34.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:59:34.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:59:34.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:59:34.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:59:34.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:59:34.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:59:34.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:59:34.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:59:34.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:59:34.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:59:34.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:59:34.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:59:34.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:59:34.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:59:34.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:59:34.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:59:34.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-17T16:59:34.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-17T16:59:34.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-17T16:59:34.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-17T16:59:34.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T16:59:34.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:59:34.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T16:59:34.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:59:34.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:59:34.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:59:34.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:59:34.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:59:34.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:59:34.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:59:34.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:59:34.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:59:34.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:59:34.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:59:34.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:59:34.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:59:34.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:59:34.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:59:34.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:59:34.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:59:34.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:59:34.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T16:59:34.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:59:34.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:59:34.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:59:34.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T16:59:34.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:59:34.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:59:34.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-17T16:59:34.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:59:34.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:59:34.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:59:34.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.87940 2024-09-17T16:59:34.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:59:34.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T16:59:34.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:59:34.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-17T16:59:34.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-17T16:59:34.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-17T16:59:34.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:59:34.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:59:34.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:59:34.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-17T16:59:34.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-17T16:59:34.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:59:34.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:59:34.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:59:34.785 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:59:34.785 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:59:34.785 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:59:34.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:59:34.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T16:59:34.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:59:34.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:59:34.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:59:34.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:59:34.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:59:34.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:59:34.955 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:59:34.955 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:59:34.955 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:59:34.955 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:59:34.955 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:59:34.955 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:59:34.955 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:59:34.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:59:34.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get fsid 2024-09-17T16:59:35.034 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:59:35.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:59:35.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:59:35.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:59:35.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:59:35.035 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:59:35.035 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:59:35.035 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:59:35.035 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:59:35.035 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:59:35.035 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:59:35.035 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:59:35.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:59:35.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get mon_host 2024-09-17T16:59:35.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-17T16:59:35.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-17T16:59:35.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:59:35.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:59:35.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:59:35.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-17T16:59:35.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:59:35.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:59:35.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:59:35.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:59:35.976 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:59:35.976 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:59:35.976 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:59:35.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:59:35.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:59:35.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:59:36.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-17T16:59:36.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T16:59:36.285 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T16:59:36.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T16:59:37.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-17T16:59:37.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:59:37.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:59:37.297 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:59:37.298 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:59:37.298 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:59:37.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T16:59:37.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-17T16:59:37.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-mon.a.asok log flush 2024-09-17T16:59:37.372 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T16:59:37.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-17T16:59:37.381 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:59:34.816+0000 7f08fd727c80 0 load: jerasure load: lrc 2024-09-17T16:59:37.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_1 td/test-erasure-eio 2024-09-17T16:59:37.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:287: TEST_rados_get_subread_eio_shard_1: local dir=td/test-erasure-eio 2024-09-17T16:59:37.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:288: TEST_rados_get_subread_eio_shard_1: setup_osds 4 2024-09-17T16:59:37.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-17T16:59:37.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-17T16:59:37.382 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-17T16:59:37.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-17T16:59:37.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T16:59:37.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-17T16:59:37.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T16:59:37.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:59:37.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:59:37.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:59:37.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-17T16:59:37.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T16:59:37.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:59:37.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:59:37.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:59:37.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-17T16:59:37.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-17T16:59:37.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:59:37.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:59:37.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T16:59:37.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:59:37.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:59:37.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:59:37.389 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:59:37.389 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:59:37.389 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T16:59:37.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:59:37.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T16:59:37.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:59:37.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:59:37.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:59:37.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T16:59:37.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T16:59:37.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:59:37.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:59:37.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:59:37.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:59:37.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:59:37.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:59:37.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-17T16:59:37.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:59:37.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=398d96f7-0620-4876-83c8-89bef4ccdf57 2024-09-17T16:59:37.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 398d96f7-0620-4876-83c8-89bef4ccdf57' 2024-09-17T16:59:37.391 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 398d96f7-0620-4876-83c8-89bef4ccdf57 2024-09-17T16:59:37.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:59:37.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB5telmZdETGBAA9P4tooWzdqU2nmUELkP7jA== 2024-09-17T16:59:37.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB5telmZdETGBAA9P4tooWzdqU2nmUELkP7jA=="}' 2024-09-17T16:59:37.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 398d96f7-0620-4876-83c8-89bef4ccdf57 -i td/test-erasure-eio/0/new.json 2024-09-17T16:59:37.661 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:59:37.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-17T16:59:37.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5telmZdETGBAA9P4tooWzdqU2nmUELkP7jA== --osd-uuid 398d96f7-0620-4876-83c8-89bef4ccdf57 2024-09-17T16:59:37.701 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:59:37.698+0000 7f4d48c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:59:37.730 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:59:37.727+0000 7f4d48c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:59:37.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:59:37.728+0000 7f4d48c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:59:37.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:59:37.728+0000 7f4d48c0e640 -1 bdev(0x557739c73c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:59:37.732 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:59:37.728+0000 7f4d48c0e640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-17T16:59:43.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-17T16:59:43.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:59:43.700 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T16:59:43.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:59:43.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:59:44.243 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T16:59:44.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:59:44.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:59:44.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:59:44.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:59:44.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:59:44.270 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:59:44.267+0000 7f090961c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:59:44.292 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:59:44.288+0000 7f090961c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:59:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:59:44.290+0000 7f090961c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:59:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:59:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:59:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:59:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:59:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:59:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:59:44.740 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T16:59:44.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:44.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:59:45.159 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:59:45.156+0000 7f090961c640 -1 Falling back to public interface 2024-09-17T16:59:45.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:46.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:46.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:46.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:59:46.166 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T16:59:46.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:46.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:59:46.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:47.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:47.600 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T16:59:47.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:47.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:59:47.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:47.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:59:47.776 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T16:59:47.773+0000 7f090961c640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:59:48.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:49.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:49.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:49.020 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T16:59:49.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:59:49.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:49.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:59:49.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:50.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:50.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:50.493 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T16:59:50.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:59:50.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:50.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:59:50.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:51.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:51.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:51.905 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T16:59:51.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:59:51.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:51.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:59:52.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:53.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:53.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:53.307 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T16:59:53.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:59:53.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:53.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:59:53.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:54.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:54.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:54.825 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T16:59:54.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:59:54.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:54.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:59:55.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:56.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:56.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:56.246 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T16:59:56.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:59:56.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:56.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:59:56.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:57.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:57.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:57.660 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T16:59:57.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:59:57.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:57.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:59:58.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:59.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:59.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:59.057 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T16:59:59.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:59:59.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:59.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:59:59.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:00.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:00.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:00.462 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:00:00.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:00:00.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:00.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:00:01.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:02.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:02.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:02.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:00:02.197 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:00:02.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:02.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:00:03.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:04.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:04.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:04.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:00:04.369 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:00:04.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:04.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:00:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:05.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:05.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:05.976 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:00:05.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:00:05.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:05.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:00:06.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:07.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:07.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:07.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:00:07.457 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:00:07.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:07.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:00:08.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:09.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:09.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:09.165 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:00:09.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:00:09.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:09.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:00:09.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:10.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:10.939 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:00:10.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:10.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:00:10.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:10.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:00:11.439 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2841047904,v1:127.0.0.1:6803/2841047904] [v2:127.0.0.1:6804/2841047904,v1:127.0.0.1:6805/2841047904] exists,up 398d96f7-0620-4876-83c8-89bef4ccdf57 2024-09-17T17:00:11.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:00:11.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:00:11.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:00:11.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T17:00:11.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-17T17:00:11.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T17:00:11.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:00:11.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T17:00:11.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:00:11.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T17:00:11.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:00:11.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:00:11.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:00:11.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:00:11.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T17:00:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T17:00:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:00:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:00:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:00:11.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:00:11.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:00:11.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:00:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:00:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:00:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:00:11.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:00:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:00:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:00:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:00:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:00:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:00:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:00:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:00:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:00:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:00:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:00:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:00:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:00:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T17:00:11.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:00:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f32f100d-16be-447e-9b2e-6c6136f09084 2024-09-17T17:00:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f32f100d-16be-447e-9b2e-6c6136f09084' 2024-09-17T17:00:11.447 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 f32f100d-16be-447e-9b2e-6c6136f09084 2024-09-17T17:00:11.447 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:00:11.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCbtelmTqpqGxAA/4IMroictXlyTEkwkm8jdQ== 2024-09-17T17:00:11.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCbtelmTqpqGxAA/4IMroictXlyTEkwkm8jdQ=="}' 2024-09-17T17:00:11.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f32f100d-16be-447e-9b2e-6c6136f09084 -i td/test-erasure-eio/1/new.json 2024-09-17T17:00:11.908 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:00:11.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-17T17:00:11.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCbtelmTqpqGxAA/4IMroictXlyTEkwkm8jdQ== --osd-uuid f32f100d-16be-447e-9b2e-6c6136f09084 2024-09-17T17:00:11.949 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:11.947+0000 7f438a20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:00:11.965 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:11.963+0000 7f438a20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:00:11.966 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:11.964+0000 7f438a20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:00:11.966 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:11.964+0000 7f438a20c640 -1 bdev(0x558d209cfc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-17T17:00:11.966 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:11.964+0000 7f438a20c640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-17T17:00:17.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-17T17:00:17.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:00:17.895 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T17:00:17.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T17:00:17.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:00:18.448 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T17:00:18.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T17:00:18.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:00:18.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:00:18.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:00:18.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:00:18.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:18.473+0000 7f16ab60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:00:18.491 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:18.489+0000 7f16ab60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:00:18.492 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:18.490+0000 7f16ab60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:00:18.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T17:00:18.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:00:18.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T17:00:18.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:00:18.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:00:18.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:18.898 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:00:18.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:00:18.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:18.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:19.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:19.353 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:19.351+0000 7f16ab60b640 -1 Falling back to public interface 2024-09-17T17:00:20.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:20.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:20.324 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:00:20.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:00:20.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:20.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:20.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:21.752 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:21.749+0000 7f16ab60b640 -1 osd.1 0 log_to_monitors true 2024-09-17T17:00:21.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:21.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:21.759 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:00:21.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:00:21.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:21.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:22.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:23.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:23.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:23.229 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:00:23.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:00:23.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:23.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:23.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:24.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:24.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:24.688 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:00:24.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:00:24.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:24.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:25.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:26.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:26.220 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T17:00:26.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:26.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:00:26.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:26.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:26.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:27.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:27.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:27.830 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T17:00:27.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:00:27.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:27.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:28.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:29.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:29.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:29.252 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T17:00:29.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:00:29.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:29.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:29.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:30.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:30.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:30.662 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T17:00:30.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:00:30.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:30.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:31.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:32.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:32.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:32.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:00:32.136 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T17:00:32.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:32.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:32.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:33.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:33.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:33.552 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T17:00:33.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:00:33.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:33.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:34.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:35.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:35.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:35.090 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:00:35.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:00:35.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:35.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:35.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:36.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:36.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:36.936 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:00:36.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:00:36.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:36.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:37.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:38.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:38.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:38.622 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:00:38.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:00:38.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:38.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:39.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:40.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:40.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:40.551 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:00:40.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:00:40.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:40.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:41.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:42.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:42.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:42.321 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:00:42.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:00:42.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:42.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:43.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:44.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:44.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:44.039 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:00:44.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:00:44.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:44.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:44.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:45.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:45.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:45.502 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:00:45.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:00:45.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:45.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:00:45.948 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/669113345,v1:127.0.0.1:6811/669113345] [v2:127.0.0.1:6812/669113345,v1:127.0.0.1:6813/669113345] exists,up f32f100d-16be-447e-9b2e-6c6136f09084 2024-09-17T17:00:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:00:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:00:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:00:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T17:00:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-17T17:00:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T17:00:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:00:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T17:00:45.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:00:45.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T17:00:45.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:00:45.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:00:45.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:00:45.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:00:45.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T17:00:45.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T17:00:45.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:00:45.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:00:45.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:00:45.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:00:45.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:00:45.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:00:45.954 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:00:45.954 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:00:45.954 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:00:45.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:00:45.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:00:45.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:00:45.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:00:45.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:00:45.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:00:45.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:00:45.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:00:45.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:00:45.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:00:45.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:00:45.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:00:45.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:00:45.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T17:00:45.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:00:45.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=51adb779-dd05-4f5b-be0d-059b0d39249f 2024-09-17T17:00:45.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 51adb779-dd05-4f5b-be0d-059b0d39249f' 2024-09-17T17:00:45.955 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 51adb779-dd05-4f5b-be0d-059b0d39249f 2024-09-17T17:00:45.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:00:45.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC9telmSRjLORAAMW1iArsu/z8NCNMKUPmf0g== 2024-09-17T17:00:45.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC9telmSRjLORAAMW1iArsu/z8NCNMKUPmf0g=="}' 2024-09-17T17:00:45.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 51adb779-dd05-4f5b-be0d-059b0d39249f -i td/test-erasure-eio/2/new.json 2024-09-17T17:00:46.476 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:00:46.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-17T17:00:46.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9telmSRjLORAAMW1iArsu/z8NCNMKUPmf0g== --osd-uuid 51adb779-dd05-4f5b-be0d-059b0d39249f 2024-09-17T17:00:46.517 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:46.515+0000 7fc587c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:00:46.540 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:46.538+0000 7fc587c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:00:46.541 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:46.539+0000 7fc587c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:00:46.542 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:46.540+0000 7fc587c14640 -1 bdev(0x559b14120800 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-17T17:00:46.542 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:46.540+0000 7fc587c14640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-17T17:00:52.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-17T17:00:52.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:00:52.493 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T17:00:52.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T17:00:52.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:00:53.039 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T17:00:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T17:00:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:00:53.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:00:53.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:00:53.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:00:53.067 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:53.065+0000 7fb60aef1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:00:53.075 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:53.073+0000 7fb60aef1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:00:53.076 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:53.074+0000 7fb60aef1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:00:53.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T17:00:53.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:00:53.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T17:00:53.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:00:53.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:00:53.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:53.454 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:00:53.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:00:53.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:53.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:00:53.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:54.419 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:54.417+0000 7fb60aef1640 -1 Falling back to public interface 2024-09-17T17:00:54.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:54.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:54.884 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:00:54.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:00:54.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:54.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:00:55.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:56.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:56.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:56.311 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:00:56.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:00:56.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:56.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:00:56.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:57.588 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:00:57.586+0000 7fb60aef1640 -1 osd.2 0 log_to_monitors true 2024-09-17T17:00:57.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:57.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:57.729 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:00:57.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:00:57.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:57.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:00:58.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:00:59.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:00:59.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:00:59.342 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:00:59.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:00:59.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:00:59.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:00:59.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:00.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:00.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:00.744 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T17:01:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:01:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:01:01.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:02.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:02.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:02.153 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T17:01:02.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:01:02.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:02.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:01:02.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:03.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:03.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:03.569 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T17:01:03.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:01:03.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:03.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:01:03.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:04.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:04.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:04.968 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T17:01:04.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:01:04.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:04.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:01:05.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:06.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:06.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:06.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:01:06.513 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T17:01:06.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:06.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:01:06.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:07.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:07.970 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T17:01:07.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:07.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:01:07.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:07.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:01:08.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:09.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:09.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:09.477 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:01:09.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:01:09.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:09.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:01:09.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:10.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:10.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:10.963 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:01:10.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:01:10.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:10.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:01:11.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:12.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:12.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:12.676 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:01:12.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:01:12.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:12.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:01:13.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:14.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:14.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:14.677 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:01:14.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:01:14.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:14.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:01:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:16.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:16.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:16.521 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:01:16.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:01:16.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:16.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:01:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:18.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:18.307 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:01:18.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:18.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:01:18.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:18.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:01:19.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:20.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:20.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:20.177 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:01:20.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:01:20.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:20.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:01:20.720 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2968031412,v1:127.0.0.1:6819/2968031412] [v2:127.0.0.1:6820/2968031412,v1:127.0.0.1:6821/2968031412] exists,up 51adb779-dd05-4f5b-be0d-059b0d39249f 2024-09-17T17:01:20.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:01:20.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:01:20.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:01:20.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T17:01:20.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-17T17:01:20.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T17:01:20.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:01:20.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T17:01:20.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:01:20.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T17:01:20.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:01:20.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:01:20.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:01:20.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:01:20.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T17:01:20.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T17:01:20.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:01:20.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:01:20.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:01:20.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:01:20.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:01:20.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:01:20.727 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:01:20.727 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:01:20.727 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:01:20.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:01:20.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:01:20.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:01:20.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:01:20.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:01:20.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:01:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:01:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:01:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:01:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:01:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:01:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:01:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:01:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T17:01:20.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:01:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5e5d65ed-0934-41fb-b0bd-6c440a628a11 2024-09-17T17:01:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 5e5d65ed-0934-41fb-b0bd-6c440a628a11' 2024-09-17T17:01:20.729 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 5e5d65ed-0934-41fb-b0bd-6c440a628a11 2024-09-17T17:01:20.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:01:20.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDgtelm72M4LBAA5Qr7XWfiGktexpL/mTeWUA== 2024-09-17T17:01:20.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDgtelm72M4LBAA5Qr7XWfiGktexpL/mTeWUA=="}' 2024-09-17T17:01:20.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5e5d65ed-0934-41fb-b0bd-6c440a628a11 -i td/test-erasure-eio/3/new.json 2024-09-17T17:01:21.186 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:01:21.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-17T17:01:21.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDgtelm72M4LBAA5Qr7XWfiGktexpL/mTeWUA== --osd-uuid 5e5d65ed-0934-41fb-b0bd-6c440a628a11 2024-09-17T17:01:21.228 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:01:21.226+0000 7fdb7ef34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:01:21.388 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:01:21.385+0000 7fdb7ef34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:01:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:01:21.386+0000 7fdb7ef34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:01:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:01:21.386+0000 7fdb7ef34640 -1 bdev(0x55de00e43c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-17T17:01:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:01:21.386+0000 7fdb7ef34640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-17T17:01:28.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-17T17:01:28.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:01:28.356 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-17T17:01:28.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T17:01:28.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:01:28.897 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T17:01:28.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T17:01:28.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:01:28.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:01:28.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:01:28.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:01:28.924 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:01:28.922+0000 7f08fea14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:01:28.965 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:01:28.963+0000 7f08fea14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:01:28.966 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:01:28.964+0000 7f08fea14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:01:29.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T17:01:29.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:01:29.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T17:01:29.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:01:29.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:01:29.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:29.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:01:29.322 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:01:29.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:29.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:29.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:30.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:01:30.559+0000 7f08fea14640 -1 Falling back to public interface 2024-09-17T17:01:30.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:30.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:30.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:01:30.740 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:01:30.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:30.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:31.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:32.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:32.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:32.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:01:32.167 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:01:32.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:32.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:32.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:33.395 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:01:33.393+0000 7f08fea14640 -1 osd.3 0 log_to_monitors true 2024-09-17T17:01:33.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:33.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:33.593 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:01:33.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:01:33.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:33.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:34.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:35.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:35.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:35.110 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:01:35.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:01:35.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:35.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:35.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:36.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:36.589 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T17:01:36.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:36.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:01:36.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:36.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:37.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:38.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:38.165 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T17:01:38.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:38.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:01:38.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:38.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:38.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:39.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:39.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:39.696 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T17:01:39.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:01:39.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:39.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:40.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:41.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:41.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:41.236 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T17:01:41.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:01:41.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:41.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:41.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:42.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:42.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:42.650 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T17:01:42.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:01:42.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:42.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:43.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:44.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:44.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:44.195 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T17:01:44.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:01:44.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:44.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:44.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:45.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:45.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:45.721 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:01:45.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:01:45.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:45.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:46.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:47.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:47.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:47.318 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:01:47.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:01:47.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:47.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:47.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:48.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:48.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:48.798 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:01:48.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:01:48.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:48.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:49.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:50.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:50.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:50.929 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:01:50.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:01:50.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:50.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:51.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:52.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:52.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:52.820 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:01:52.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:01:52.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:52.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:53.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:54.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:54.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:54.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:01:54.772 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:01:54.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:54.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:55.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:56.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:56.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:56.674 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:01:56.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:01:56.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:56.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:57.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:58.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:58.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:58.250 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T17:01:58.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T17:01:58.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:58.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:01:58.734 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/437181100,v1:127.0.0.1:6827/437181100] [v2:127.0.0.1:6828/437181100,v1:127.0.0.1:6829/437181100] exists,up 5e5d65ed-0934-41fb-b0bd-6c440a628a11 2024-09-17T17:01:58.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:01:58.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:01:58.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:01:58.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-17T17:01:58.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T17:01:58.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T17:01:58.736 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:01:58.736 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:01:58.736 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:01:58.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T17:01:58.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-17T17:01:58.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.0.asok log flush 2024-09-17T17:01:58.815 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T17:01:58.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-17T17:01:58.825 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T16:59:45.160+0000 7f090961c640 0 load: jerasure load: lrc 2024-09-17T17:01:58.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:290: TEST_rados_get_subread_eio_shard_1: local poolname=pool-jerasure 2024-09-17T17:01:58.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:291: TEST_rados_get_subread_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-17T17:01:58.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T17:01:58.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-17T17:01:58.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-17T17:01:58.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-17T17:01:58.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-17T17:01:58.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-17T17:01:58.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-17T17:01:59.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-17T17:01:59.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-17T17:02:00.561 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-17T17:02:00.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T17:02:01.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-17T17:02:01.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T17:02:01.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T17:02:01.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T17:02:01.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T17:02:01.576 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T17:02:01.576 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T17:02:01.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T17:02:01.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T17:02:01.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T17:02:01.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T17:02:01.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T17:02:01.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T17:02:01.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T17:02:01.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T17:02:01.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T17:02:02.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T17:02:02.070 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:02:02.070 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T17:02:02.071 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T17:02:02.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T17:02:02.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:02:02.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T17:02:02.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803805 2024-09-17T17:02:02.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803805 2024-09-17T17:02:02.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805' 2024-09-17T17:02:02.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:02:02.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T17:02:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574871 2024-09-17T17:02:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574871 2024-09-17T17:02:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805 1-55834574871' 2024-09-17T17:02:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:02:02.440 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T17:02:02.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345935 2024-09-17T17:02:02.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345935 2024-09-17T17:02:02.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805 1-55834574871 2-85899345935' 2024-09-17T17:02:02.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:02:02.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T17:02:02.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018889 2024-09-17T17:02:02.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018889 2024-09-17T17:02:02.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805 1-55834574871 2-85899345935 3-128849018889' 2024-09-17T17:02:02.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:02:02.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803805 2024-09-17T17:02:02.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:02:02.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T17:02:02.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803805 2024-09-17T17:02:02.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:02:02.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803805 2024-09-17T17:02:02.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803805' 2024-09-17T17:02:02.668 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803805 2024-09-17T17:02:02.668 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:02:03.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803803 -lt 25769803805 2024-09-17T17:02:03.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T17:02:04.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T17:02:04.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:02:04.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803805 -lt 25769803805 2024-09-17T17:02:04.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:02:04.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574871 2024-09-17T17:02:04.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:02:04.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T17:02:04.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574871 2024-09-17T17:02:04.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:02:04.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574871 2024-09-17T17:02:04.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574871' 2024-09-17T17:02:04.496 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574871 2024-09-17T17:02:04.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T17:02:04.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574871 -lt 55834574871 2024-09-17T17:02:04.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:02:04.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345935 2024-09-17T17:02:04.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:02:04.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T17:02:04.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345935 2024-09-17T17:02:04.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:02:04.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345935 2024-09-17T17:02:04.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345935' 2024-09-17T17:02:04.922 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 85899345935 2024-09-17T17:02:04.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T17:02:05.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345935 -lt 85899345935 2024-09-17T17:02:05.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:02:05.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018889 2024-09-17T17:02:05.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:02:05.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T17:02:05.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018889 2024-09-17T17:02:05.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:02:05.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018889 2024-09-17T17:02:05.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018889' 2024-09-17T17:02:05.348 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 128849018889 2024-09-17T17:02:05.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T17:02:05.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018890 -lt 128849018889 2024-09-17T17:02:05.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T17:02:05.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:02:05.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:02:06.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T17:02:06.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T17:02:06.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T17:02:06.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T17:02:06.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T17:02:06.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T17:02:06.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T17:02:06.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T17:02:06.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T17:02:06.687 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T17:02:06.687 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:02:06.687 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:02:07.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T17:02:07.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T17:02:07.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T17:02:07.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:293: TEST_rados_get_subread_eio_shard_1: local shard_id=1 2024-09-17T17:02:07.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:294: TEST_rados_get_subread_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 2024-09-17T17:02:07.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-17T17:02:07.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-17T17:02:07.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-17T17:02:07.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-17T17:02:07.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-17T17:02:07.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-17T17:02:07.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-17T17:02:07.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-17T17:02:07.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-87940-1 2024-09-17T17:02:07.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-87940-1 2024-09-17T17:02:07.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T17:02:07.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T17:02:07.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-87940-1 2024-09-17T17:02:07.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:02:07.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T17:02:07.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:02:07.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T17:02:07.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:02:07.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T17:02:07.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:02:07.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T17:02:07.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-87940-1 td/test-erasure-eio/ORIGINAL 2024-09-17T17:02:07.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-87940-1 td/test-erasure-eio 1 2024-09-17T17:02:07.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T17:02:07.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T17:02:07.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T17:02:07.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T17:02:07.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T17:02:07.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T17:02:07.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-87940-1 2024-09-17T17:02:07.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T17:02:07.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T17:02:07.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T17:02:07.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-17T17:02:07.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T17:02:07.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-87940-1 2024-09-17T17:02:07.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T17:02:07.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-87940-1 2024-09-17T17:02:07.377 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-87940-1 2024-09-17T17:02:07.377 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T17:02:07.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T17:02:07.794 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:02:07.794 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T17:02:07.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T17:02:07.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-17T17:02:07.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T17:02:07.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-17T17:02:07.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T17:02:07.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/1/type 2024-09-17T17:02:07.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T17:02:07.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-17T17:02:07.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T17:02:07.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-17T17:02:07.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T17:02:07.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T17:02:07.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T17:02:07.799 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-17T17:02:07.800 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T17:02:07.800 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T17:02:07.800 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:02:07.800 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:02:07.800 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:02:07.800 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.1.asok 2024-09-17T17:02:07.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-17T17:02:07.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T17:02:07.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T17:02:07.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T17:02:07.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-17T17:02:07.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T17:02:07.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T17:02:07.886 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:02:07.886 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:02:07.886 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:02:07.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.1.asok 2024-09-17T17:02:07.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T17:02:07.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-87940-1 1 2024-09-17T17:02:07.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-87940-1 2024-09-17T17:02:07.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T17:02:07.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T17:02:07.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-87940-1 2024-09-17T17:02:07.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T17:02:07.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T17:02:07.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-87940-1 td/test-erasure-eio/COPY 2024-09-17T17:02:08.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T17:02:08.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T17:02:08.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-17T17:02:08.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2024-09-17T17:02:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: local another_shard_id=2 2024-09-17T17:02:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-87940-1 td/test-erasure-eio 2 2024-09-17T17:02:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T17:02:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T17:02:08.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T17:02:08.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T17:02:08.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T17:02:08.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T17:02:08.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-87940-1 2024-09-17T17:02:08.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T17:02:08.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T17:02:08.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T17:02:08.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=2 2024-09-17T17:02:08.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T17:02:08.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-87940-1 2024-09-17T17:02:08.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T17:02:08.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-87940-1 2024-09-17T17:02:08.011 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-87940-1 2024-09-17T17:02:08.011 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T17:02:08.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T17:02:08.430 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:02:08.430 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T17:02:08.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T17:02:08.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-17T17:02:08.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T17:02:08.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=2 2024-09-17T17:02:08.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T17:02:08.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/2/type 2024-09-17T17:02:08.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T17:02:08.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-09-17T17:02:08.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T17:02:08.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=2 2024-09-17T17:02:08.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T17:02:08.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T17:02:08.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T17:02:08.433 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.2 2024-09-17T17:02:08.433 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-17T17:02:08.433 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-17T17:02:08.433 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:02:08.433 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:02:08.433 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:02:08.433 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.2.asok 2024-09-17T17:02:08.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-09-17T17:02:08.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T17:02:08.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T17:02:08.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T17:02:08.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.2 2024-09-17T17:02:08.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-17T17:02:08.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-17T17:02:08.521 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:02:08.521 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:02:08.521 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:02:08.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.2.asok 2024-09-17T17:02:08.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T17:02:08.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.2.asok injectdataerr pool-jerasure obj-eio-87940-1 2 2024-09-17T17:02:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: '[' 1 -eq 1 -a 2 -eq 2 ']' 2024-09-17T17:02:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:188: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-87940-1 2024-09-17T17:02:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T17:02:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T17:02:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-87940-1 2024-09-17T17:02:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T17:02:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T17:02:08.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-87940-1 td/test-erasure-eio/COPY 2024-09-17T17:02:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T17:02:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T17:02:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-17T17:02:08.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:295: TEST_rados_get_subread_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-17T17:02:08.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T17:02:08.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-17T17:02:09.124 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-17T17:02:09.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-17T17:02:09.771 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-17T17:02:09.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-17T17:02:09.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T17:02:09.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:02:09.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T17:02:09.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:02:09.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:02:09.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:02:09.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:02:09.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:02:09.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:02:09.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:02:09.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:02:09.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:02:09.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:02:09.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:02:09.895 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:02:09.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:02:09.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:02:09.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:02:09.896 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:02:09.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:02:09.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:02:09.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T17:02:09.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:02:09.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:02:09.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:02:09.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T17:02:09.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:02:09.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:02:09.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-17T17:02:09.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-17T17:02:09.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-17T17:02:09.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-17T17:02:09.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T17:02:09.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:02:09.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T17:02:09.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:02:09.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:02:09.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:02:09.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:02:09.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:02:09.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:02:09.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:02:09.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:02:09.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:02:09.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:02:09.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:02:09.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:02:09.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:02:09.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:02:09.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:02:09.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:02:09.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:02:09.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:02:09.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T17:02:09.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:02:09.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:02:09.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:02:09.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T17:02:09.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:02:09.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:02:09.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-17T17:02:09.930 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T17:02:09.930 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:02:09.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:02:09.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.87940 2024-09-17T17:02:09.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T17:02:09.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T17:02:09.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T17:02:09.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-17T17:02:09.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-17T17:02:09.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-17T17:02:09.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T17:02:09.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T17:02:09.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T17:02:09.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-17T17:02:09.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-17T17:02:10.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T17:02:10.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:02:10.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:02:10.388 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:02:10.388 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:02:10.388 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:02:10.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:02:10.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T17:02:10.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T17:02:10.581 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T17:02:10.581 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:02:10.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T17:02:10.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T17:02:10.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T17:02:10.583 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T17:02:10.583 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T17:02:10.583 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T17:02:10.583 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:02:10.583 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:02:10.583 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:02:10.583 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T17:02:10.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:02:10.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get fsid 2024-09-17T17:02:10.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T17:02:10.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:02:10.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T17:02:10.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T17:02:10.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T17:02:10.663 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T17:02:10.663 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T17:02:10.663 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T17:02:10.663 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:02:10.664 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:02:10.664 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:02:10.664 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T17:02:10.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:02:10.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get mon_host 2024-09-17T17:02:10.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-17T17:02:10.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-17T17:02:10.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T17:02:10.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T17:02:10.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T17:02:10.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-17T17:02:10.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T17:02:11.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T17:02:11.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:02:11.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:02:11.596 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:02:11.596 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:02:11.597 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:02:11.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:02:11.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:02:11.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:02:11.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-17T17:02:11.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T17:02:11.913 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T17:02:11.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T17:02:12.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-17T17:02:12.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T17:02:12.926 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T17:02:12.926 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:02:12.926 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:02:12.926 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:02:12.926 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T17:02:12.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-17T17:02:12.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-mon.a.asok log flush 2024-09-17T17:02:13.000 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T17:02:13.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-17T17:02:13.010 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T17:02:10.420+0000 7f5846122c80 0 load: jerasure load: lrc 2024-09-17T17:02:13.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_missing td/test-erasure-eio 2024-09-17T17:02:13.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:302: TEST_rados_get_subread_missing: local dir=td/test-erasure-eio 2024-09-17T17:02:13.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:303: TEST_rados_get_subread_missing: setup_osds 4 2024-09-17T17:02:13.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-17T17:02:13.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-17T17:02:13.011 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-17T17:02:13.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-17T17:02:13.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T17:02:13.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-17T17:02:13.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T17:02:13.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:02:13.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T17:02:13.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:02:13.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-17T17:02:13.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:02:13.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:02:13.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:02:13.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:02:13.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-17T17:02:13.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-17T17:02:13.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:02:13.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:02:13.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:02:13.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:02:13.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:02:13.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:02:13.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:02:13.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:02:13.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:02:13.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:02:13.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:02:13.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:02:13.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:02:13.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:02:13.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:02:13.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:02:13.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:02:13.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:02:13.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:02:13.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:02:13.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:02:13.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:02:13.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-17T17:02:13.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:02:13.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a4e1ceb4-be06-4575-b6ef-a8ef826eedb4 2024-09-17T17:02:13.018 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 a4e1ceb4-be06-4575-b6ef-a8ef826eedb4 2024-09-17T17:02:13.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a4e1ceb4-be06-4575-b6ef-a8ef826eedb4' 2024-09-17T17:02:13.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:02:13.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAVtulmOMMFAhAAeBFIzLSamoiXBONwOmSUJQ== 2024-09-17T17:02:13.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAVtulmOMMFAhAAeBFIzLSamoiXBONwOmSUJQ=="}' 2024-09-17T17:02:13.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a4e1ceb4-be06-4575-b6ef-a8ef826eedb4 -i td/test-erasure-eio/0/new.json 2024-09-17T17:02:13.298 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:02:13.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-17T17:02:13.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVtulmOMMFAhAAeBFIzLSamoiXBONwOmSUJQ== --osd-uuid a4e1ceb4-be06-4575-b6ef-a8ef826eedb4 2024-09-17T17:02:13.337 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:13.335+0000 7fdad994b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:13.366 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:13.364+0000 7fdad994b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:13.367 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:13.366+0000 7fdad994b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:13.368 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:13.366+0000 7fdad994b640 -1 bdev(0x55f44ba03c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-17T17:02:13.368 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:13.366+0000 7fdad994b640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-17T17:02:19.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-17T17:02:19.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:02:19.310 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T17:02:19.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T17:02:19.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:02:19.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T17:02:19.852 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T17:02:19.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:02:19.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:02:19.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:02:19.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:02:19.879 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:19.877+0000 7ff13ba0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:19.896 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:19.894+0000 7ff13ba0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:19.897 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:19.895+0000 7ff13ba0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:20.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T17:02:20.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:02:20.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T17:02:20.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:02:20.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:02:20.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:20.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:02:20.273 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:02:20.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:20.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:20.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:21.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:21.478+0000 7ff13ba0f640 -1 Falling back to public interface 2024-09-17T17:02:21.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:21.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:21.661 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:02:21.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:02:21.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:21.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:22.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:23.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:23.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:23.098 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:02:23.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:02:23.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:23.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:23.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:24.366 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:24.364+0000 7ff13ba0f640 -1 osd.0 0 log_to_monitors true 2024-09-17T17:02:24.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:24.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:24.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:02:24.531 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:02:24.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:24.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:24.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:25.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:25.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:25.987 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:02:25.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:02:25.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:25.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:26.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:27.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:27.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:27.457 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T17:02:27.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:02:27.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:27.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:27.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:28.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:28.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:28.863 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T17:02:28.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:02:28.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:28.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:29.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:30.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:30.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:30.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:02:30.260 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T17:02:30.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:30.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:30.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:31.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:31.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:31.698 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T17:02:31.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:02:31.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:31.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:32.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:33.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:33.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:33.130 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T17:02:33.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:02:33.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:33.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:33.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:34.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:34.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:34.654 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T17:02:34.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:02:34.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:34.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:34.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:35.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:35.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:35.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:02:35.943 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:02:35.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:35.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:36.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:37.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:37.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:37.531 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:02:37.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:02:37.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:37.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:38.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:39.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:39.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:39.081 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:02:39.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:02:39.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:39.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:39.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:40.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:40.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:40.843 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:02:40.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:02:40.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:40.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:41.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:42.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:42.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:42.612 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:02:42.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:02:42.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:42.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:43.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:44.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:44.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:44.290 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:02:44.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:02:44.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:44.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:45.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:46.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:46.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:46.209 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:02:46.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:02:46.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:46.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:47.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:48.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:48.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:48.072 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T17:02:48.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T17:02:48.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:48.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:02:48.501 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4227227266,v1:127.0.0.1:6803/4227227266] [v2:127.0.0.1:6804/4227227266,v1:127.0.0.1:6805/4227227266] exists,up a4e1ceb4-be06-4575-b6ef-a8ef826eedb4 2024-09-17T17:02:48.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:02:48.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:02:48.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:02:48.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T17:02:48.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-17T17:02:48.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T17:02:48.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:02:48.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T17:02:48.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:02:48.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T17:02:48.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:02:48.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:02:48.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:02:48.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:02:48.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T17:02:48.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T17:02:48.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:02:48.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:02:48.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:02:48.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:02:48.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:02:48.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:02:48.507 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:02:48.507 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:02:48.507 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:02:48.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:02:48.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:02:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:02:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:02:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:02:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:02:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:02:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:02:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:02:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:02:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:02:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:02:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:02:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T17:02:48.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:02:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=efc4e443-585d-40ad-98d5-637384df5871 2024-09-17T17:02:48.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 efc4e443-585d-40ad-98d5-637384df5871' 2024-09-17T17:02:48.509 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 efc4e443-585d-40ad-98d5-637384df5871 2024-09-17T17:02:48.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:02:48.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA4tulm+IccHxAA1hvcfuSVQMeZzJrSTOyV1g== 2024-09-17T17:02:48.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA4tulm+IccHxAA1hvcfuSVQMeZzJrSTOyV1g=="}' 2024-09-17T17:02:48.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new efc4e443-585d-40ad-98d5-637384df5871 -i td/test-erasure-eio/1/new.json 2024-09-17T17:02:49.044 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:02:49.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-17T17:02:49.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4tulm+IccHxAA1hvcfuSVQMeZzJrSTOyV1g== --osd-uuid efc4e443-585d-40ad-98d5-637384df5871 2024-09-17T17:02:49.085 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:49.082+0000 7fcdbc613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:49.087 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:49.084+0000 7fcdbc613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:49.088 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:49.085+0000 7fcdbc613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:49.089 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:49.086+0000 7fcdbc613640 -1 bdev(0x564bfa9d3c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-17T17:02:49.089 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:49.086+0000 7fcdbc613640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-17T17:02:54.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-17T17:02:54.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:02:54.891 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T17:02:54.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T17:02:54.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:02:55.430 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T17:02:55.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T17:02:55.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:02:55.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:02:55.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:02:55.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:02:55.458 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:55.455+0000 7f8cf520d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:55.472 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:55.469+0000 7f8cf520d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:55.473 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:55.470+0000 7f8cf520d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:55.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T17:02:55.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:02:55.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T17:02:55.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:02:55.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:02:55.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:55.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:02:55.870 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:02:55.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:55.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:02:56.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:57.075 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:57.072+0000 7f8cf520d640 -1 Falling back to public interface 2024-09-17T17:02:57.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:57.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:57.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:02:57.290 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:02:57.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:57.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:02:57.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:58.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:58.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:58.747 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:02:58.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:02:58.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:58.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:02:59.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:59.876 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:02:59.874+0000 7f8cf520d640 -1 osd.1 0 log_to_monitors true 2024-09-17T17:03:00.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:00.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:00.192 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:03:00.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:03:00.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:00.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:00.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:01.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:01.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:01.720 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:03:01.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:03:01.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:01.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:02.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:03.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:03.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:03.255 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T17:03:03.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:03:03.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:03.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:03.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:04.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:04.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:04.803 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T17:03:04.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:03:04.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:04.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:05.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:06.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:06.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:06.245 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T17:03:06.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:03:06.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:06.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:06.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:07.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:07.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:07.750 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T17:03:07.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:03:07.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:07.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:08.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:09.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:09.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:09.197 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T17:03:09.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:03:09.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:09.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:09.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:10.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:10.861 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T17:03:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:03:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:10.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:11.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:12.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:12.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:12.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:03:12.256 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:03:12.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:12.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:12.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:13.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:13.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:13.733 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:03:13.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:03:13.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:13.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:14.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:15.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:15.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:15.168 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:03:15.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:03:15.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:15.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:16.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:16.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:16.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:03:16.868 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:03:16.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:16.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:17.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:18.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:18.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:18.597 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:03:18.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:03:18.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:18.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:19.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:20.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:20.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:20.381 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:03:20.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:03:20.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:20.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:20.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:21.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:21.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:21.918 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:03:21.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:03:21.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:21.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:23.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:24.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:24.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:24.089 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T17:03:24.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T17:03:24.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:24.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:03:24.621 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 14 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1626261079,v1:127.0.0.1:6811/1626261079] [v2:127.0.0.1:6812/1626261079,v1:127.0.0.1:6813/1626261079] exists,up efc4e443-585d-40ad-98d5-637384df5871 2024-09-17T17:03:24.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:03:24.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:03:24.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:03:24.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T17:03:24.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-17T17:03:24.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T17:03:24.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:03:24.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T17:03:24.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:03:24.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T17:03:24.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:03:24.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:03:24.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:03:24.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:03:24.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T17:03:24.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T17:03:24.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:03:24.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:03:24.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:03:24.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:03:24.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:03:24.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:03:24.627 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:03:24.627 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:03:24.627 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:03:24.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:03:24.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:03:24.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:03:24.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:03:24.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:03:24.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:03:24.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:03:24.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:03:24.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:03:24.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:03:24.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:03:24.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:03:24.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:03:24.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T17:03:24.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:03:24.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f7f97bae-fbc9-464d-802a-1b6f3e99546e 2024-09-17T17:03:24.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 f7f97bae-fbc9-464d-802a-1b6f3e99546e' 2024-09-17T17:03:24.629 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 f7f97bae-fbc9-464d-802a-1b6f3e99546e 2024-09-17T17:03:24.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:03:24.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBctulmxOo/JhAA4wI9SVr5Y0uOuROCA41sVA== 2024-09-17T17:03:24.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBctulmxOo/JhAA4wI9SVr5Y0uOuROCA41sVA=="}' 2024-09-17T17:03:24.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f7f97bae-fbc9-464d-802a-1b6f3e99546e -i td/test-erasure-eio/2/new.json 2024-09-17T17:03:25.089 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:03:25.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-17T17:03:25.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBctulmxOo/JhAA4wI9SVr5Y0uOuROCA41sVA== --osd-uuid f7f97bae-fbc9-464d-802a-1b6f3e99546e 2024-09-17T17:03:25.130 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:03:25.128+0000 7ff3e7664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:03:25.137 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:03:25.135+0000 7ff3e7664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:03:25.138 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:03:25.136+0000 7ff3e7664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:03:25.138 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:03:25.136+0000 7ff3e7664640 -1 bdev(0x556afb56dc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-17T17:03:25.139 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:03:25.137+0000 7ff3e7664640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-17T17:03:31.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-17T17:03:31.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:03:31.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T17:03:31.123 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T17:03:31.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:03:31.664 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T17:03:31.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T17:03:31.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:03:31.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:03:31.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:03:31.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:03:31.691 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:03:31.689+0000 7f8b0bfb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:03:31.710 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:03:31.708+0000 7f8b0bfb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:03:31.712 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:03:31.710+0000 7f8b0bfb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:03:32.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T17:03:32.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:03:32.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T17:03:32.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:03:32.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:03:32.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:32.109 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:03:32.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:03:32.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:32.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:32.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:33.324 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:03:33.321+0000 7f8b0bfb0640 -1 Falling back to public interface 2024-09-17T17:03:33.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:33.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:33.536 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:03:33.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:03:33.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:33.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:33.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:34.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:34.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:34.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:03:34.963 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:03:34.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:34.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:35.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:36.233 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:03:36.230+0000 7f8b0bfb0640 -1 osd.2 0 log_to_monitors true 2024-09-17T17:03:36.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:36.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:36.356 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:03:36.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:03:36.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:36.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:36.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:37.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:37.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:37.997 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:03:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:03:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:38.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:39.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:39.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:39.509 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T17:03:39.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:03:39.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:39.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:39.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:40.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:40.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:40.903 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T17:03:40.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:03:40.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:40.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:41.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:42.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:42.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:42.397 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T17:03:42.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:03:42.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:42.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:42.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:43.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:43.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:43.796 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T17:03:43.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:03:43.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:43.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:44.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:45.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:45.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:45.210 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T17:03:45.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:03:45.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:45.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:45.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:46.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:46.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:46.642 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T17:03:46.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:03:46.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:46.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:47.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:48.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:48.100 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:03:48.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:48.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:03:48.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:48.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:48.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:49.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:49.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:49.695 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:03:49.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:03:49.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:49.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:50.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:51.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:51.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:51.575 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:03:51.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:03:51.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:51.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:52.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:53.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:53.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:53.201 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:03:53.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:03:53.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:53.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:53.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:54.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:54.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:54.948 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:03:54.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:03:54.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:54.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:55.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:56.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:56.920 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:03:56.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:56.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:03:56.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:56.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:57.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:58.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:58.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:58.787 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:03:58.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:03:58.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:58.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:03:59.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:00.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:00.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:00.593 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T17:04:00.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T17:04:00.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:00.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:04:01.181 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1987488740,v1:127.0.0.1:6819/1987488740] [v2:127.0.0.1:6820/1987488740,v1:127.0.0.1:6821/1987488740] exists,up f7f97bae-fbc9-464d-802a-1b6f3e99546e 2024-09-17T17:04:01.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:04:01.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:04:01.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:04:01.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T17:04:01.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-17T17:04:01.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T17:04:01.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:04:01.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T17:04:01.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:04:01.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T17:04:01.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:04:01.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:04:01.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:04:01.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:04:01.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T17:04:01.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T17:04:01.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:04:01.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:04:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:04:01.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:04:01.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:04:01.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:04:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:04:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:04:01.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:04:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:04:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:04:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:04:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:04:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:04:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:04:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:04:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:04:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:04:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:04:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:04:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:04:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T17:04:01.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:04:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=162dffcf-b6b0-42e2-88a5-47cebe4b6fd2 2024-09-17T17:04:01.189 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 162dffcf-b6b0-42e2-88a5-47cebe4b6fd2 2024-09-17T17:04:01.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 162dffcf-b6b0-42e2-88a5-47cebe4b6fd2' 2024-09-17T17:04:01.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:04:01.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCBtulm+SgPDBAAotTWLJxsB8w42Dxyw0me8g== 2024-09-17T17:04:01.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCBtulm+SgPDBAAotTWLJxsB8w42Dxyw0me8g=="}' 2024-09-17T17:04:01.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 162dffcf-b6b0-42e2-88a5-47cebe4b6fd2 -i td/test-erasure-eio/3/new.json 2024-09-17T17:04:02.025 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:04:02.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-17T17:04:02.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCBtulm+SgPDBAAotTWLJxsB8w42Dxyw0me8g== --osd-uuid 162dffcf-b6b0-42e2-88a5-47cebe4b6fd2 2024-09-17T17:04:02.066 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:04:02.064+0000 7fcf7562c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:04:02.068 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:04:02.066+0000 7fcf7562c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:04:02.069 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:04:02.067+0000 7fcf7562c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:04:02.069 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:04:02.068+0000 7fcf7562c640 -1 bdev(0x55e8ed2a5c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-17T17:04:02.070 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:04:02.068+0000 7fcf7562c640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-17T17:04:08.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-17T17:04:08.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:04:08.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T17:04:08.377 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-17T17:04:08.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:04:08.914 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T17:04:08.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T17:04:08.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:04:08.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:04:08.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:04:08.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:04:08.943 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:04:08.941+0000 7f6ae7444640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:04:08.951 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:04:08.949+0000 7f6ae7444640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:04:08.952 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:04:08.950+0000 7f6ae7444640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:04:09.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T17:04:09.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:04:09.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T17:04:09.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:04:09.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:04:09.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:09.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:04:09.366 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:04:09.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:09.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:09.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:10.064 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:04:10.062+0000 7f6ae7444640 -1 Falling back to public interface 2024-09-17T17:04:10.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:10.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:10.755 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:04:10.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:04:10.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:10.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:11.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:12.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:12.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:12.186 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:04:12.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:04:12.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:12.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:12.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:12.881 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:04:12.878+0000 7f6ae7444640 -1 osd.3 0 log_to_monitors true 2024-09-17T17:04:13.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:13.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:13.589 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:04:13.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:04:13.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:13.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:14.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:15.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:15.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:15.222 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:04:15.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:04:15.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:15.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:15.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:16.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:16.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:16.960 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T17:04:16.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:04:16.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:16.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:17.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:18.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:18.388 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T17:04:18.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:18.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:04:18.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:18.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:18.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:19.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:19.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:19.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:04:19.828 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T17:04:19.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:19.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:20.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:21.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:21.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:21.328 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T17:04:21.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:04:21.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:21.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:21.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:22.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:22.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:22.867 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T17:04:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:04:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:23.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:24.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:24.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:24.295 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T17:04:24.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:04:24.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:24.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:24.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:25.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:25.813 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:04:25.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:25.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:04:25.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:25.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:26.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:27.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:27.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:27.311 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:04:27.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:04:27.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:27.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:27.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:28.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:28.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:28.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:04:28.907 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:04:28.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:28.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:29.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:30.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:30.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:30.676 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:04:30.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:04:30.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:30.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:31.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:32.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:32.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:32.411 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:04:32.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:04:32.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:32.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:33.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:34.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:34.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:34.130 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:04:34.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:04:34.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:34.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:34.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:35.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:35.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:35.975 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:04:35.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:04:35.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:35.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:36.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:37.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:37.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:37.859 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T17:04:37.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T17:04:37.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:37.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:04:38.263 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3974634294,v1:127.0.0.1:6827/3974634294] [v2:127.0.0.1:6828/3974634294,v1:127.0.0.1:6829/3974634294] exists,up 162dffcf-b6b0-42e2-88a5-47cebe4b6fd2 2024-09-17T17:04:38.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:04:38.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:04:38.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:04:38.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-17T17:04:38.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T17:04:38.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T17:04:38.265 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:04:38.265 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:38.265 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:04:38.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T17:04:38.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-17T17:04:38.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.0.asok log flush 2024-09-17T17:04:38.345 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T17:04:38.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-17T17:04:38.355 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T17:02:21.483+0000 7ff13ba0f640 0 load: jerasure load: lrc 2024-09-17T17:04:38.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:305: TEST_rados_get_subread_missing: local poolname=pool-jerasure 2024-09-17T17:04:38.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:306: TEST_rados_get_subread_missing: create_erasure_coded_pool pool-jerasure 2 1 2024-09-17T17:04:38.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T17:04:38.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-17T17:04:38.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-17T17:04:38.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-17T17:04:38.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-17T17:04:38.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-17T17:04:38.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-17T17:04:39.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-17T17:04:39.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-17T17:04:40.539 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-17T17:04:40.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T17:04:41.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-17T17:04:41.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T17:04:41.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T17:04:41.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T17:04:41.553 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T17:04:41.554 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T17:04:41.554 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T17:04:41.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T17:04:41.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T17:04:41.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T17:04:41.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T17:04:41.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T17:04:41.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T17:04:41.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T17:04:41.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T17:04:41.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T17:04:42.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T17:04:42.068 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:04:42.068 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T17:04:42.068 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T17:04:42.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T17:04:42.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:04:42.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T17:04:42.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803808 2024-09-17T17:04:42.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803808 2024-09-17T17:04:42.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803808' 2024-09-17T17:04:42.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:04:42.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T17:04:42.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=60129542171 2024-09-17T17:04:42.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 60129542171 2024-09-17T17:04:42.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803808 1-60129542171' 2024-09-17T17:04:42.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:04:42.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T17:04:42.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345939 2024-09-17T17:04:42.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345939 2024-09-17T17:04:42.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803808 1-60129542171 2-85899345939' 2024-09-17T17:04:42.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:04:42.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T17:04:42.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018885 2024-09-17T17:04:42.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018885 2024-09-17T17:04:42.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803808 1-60129542171 2-85899345939 3-128849018885' 2024-09-17T17:04:42.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:04:42.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803808 2024-09-17T17:04:42.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:04:42.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T17:04:42.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803808 2024-09-17T17:04:42.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:04:42.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803808 2024-09-17T17:04:42.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2024-09-17T17:04:42.563 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803808 2024-09-17T17:04:42.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:04:42.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803806 -lt 25769803808 2024-09-17T17:04:42.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T17:04:43.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T17:04:43.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:04:44.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803809 -lt 25769803808 2024-09-17T17:04:44.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:04:44.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-60129542171 2024-09-17T17:04:44.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:04:44.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T17:04:44.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-60129542171 2024-09-17T17:04:44.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:04:44.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=60129542171 2024-09-17T17:04:44.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 60129542171' 2024-09-17T17:04:44.369 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 60129542171 2024-09-17T17:04:44.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T17:04:44.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 60129542171 -lt 60129542171 2024-09-17T17:04:44.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:04:44.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345939 2024-09-17T17:04:44.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:04:44.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T17:04:44.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345939 2024-09-17T17:04:44.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:04:44.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345939 2024-09-17T17:04:44.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345939' 2024-09-17T17:04:44.761 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 85899345939 2024-09-17T17:04:44.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T17:04:45.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345939 -lt 85899345939 2024-09-17T17:04:45.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:04:45.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018885 2024-09-17T17:04:45.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:04:45.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T17:04:45.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018885 2024-09-17T17:04:45.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:04:45.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018885 2024-09-17T17:04:45.184 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 128849018885 2024-09-17T17:04:45.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018885' 2024-09-17T17:04:45.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T17:04:45.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018886 -lt 128849018885 2024-09-17T17:04:45.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T17:04:45.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:04:45.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:04:46.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T17:04:46.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T17:04:46.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T17:04:46.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T17:04:46.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T17:04:46.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T17:04:46.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T17:04:46.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T17:04:46.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T17:04:46.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T17:04:46.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:04:46.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:04:47.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T17:04:47.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T17:04:47.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T17:04:47.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:308: TEST_rados_get_subread_missing: local shard_id=1 2024-09-17T17:04:47.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:309: TEST_rados_get_subread_missing: rados_put_get_data remove td/test-erasure-eio 1 2024-09-17T17:04:47.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=remove 2024-09-17T17:04:47.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-17T17:04:47.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-17T17:04:47.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-17T17:04:47.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-17T17:04:47.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-17T17:04:47.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-17T17:04:47.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-17T17:04:47.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-remove-87940-1 2024-09-17T17:04:47.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-remove-87940-1 2024-09-17T17:04:47.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T17:04:47.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T17:04:47.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-remove-87940-1 2024-09-17T17:04:47.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:04:47.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T17:04:47.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:04:47.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T17:04:47.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:04:47.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T17:04:47.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:04:47.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T17:04:47.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-remove-87940-1 td/test-erasure-eio/ORIGINAL 2024-09-17T17:04:47.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-87940-1 td/test-erasure-eio 1 2024-09-17T17:04:47.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2024-09-17T17:04:47.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2024-09-17T17:04:47.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2024-09-17T17:04:47.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2024-09-17T17:04:47.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2024-09-17T17:04:47.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2024-09-17T17:04:47.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-87940-1 2024-09-17T17:04:47.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2024-09-17T17:04:47.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2024-09-17T17:04:47.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2024-09-17T17:04:47.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=1 2024-09-17T17:04:47.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2024-09-17T17:04:47.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-87940-1 2024-09-17T17:04:47.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T17:04:47.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-remove-87940-1 2024-09-17T17:04:47.219 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-remove-87940-1 2024-09-17T17:04:47.220 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T17:04:47.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T17:04:47.645 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:04:47.645 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T17:04:47.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T17:04:47.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2024-09-17T17:04:47.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2024-09-17T17:04:47.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=1 2024-09-17T17:04:47.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 1 obj-remove-87940-1 remove 2024-09-17T17:04:47.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T17:04:47.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T17:04:47.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T17:04:47.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T17:04:47.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-remove-87940-1 remove 2024-09-17T17:04:47.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T17:04:47.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T17:04:47.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T17:04:47.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T17:04:47.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-17T17:04:47.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:04:47.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:04:47.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:04:47.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:04:47.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:04:47.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:04:47.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-remove-87940-1 remove 2024-09-17T17:04:47.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T17:04:47.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T17:04:47.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T17:04:47.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T17:04:47.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-17T17:04:47.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-remove-87940-1 remove 2024-09-17T17:04:50.092 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#2:7040a0dd:::obj-remove-87940-1:head# 2024-09-17T17:04:50.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-17T17:04:50.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T17:04:50.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T17:04:50.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T17:04:50.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T17:04:50.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T17:04:50.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:04:50.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:04:50.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:04:50.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:04:50.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T17:04:50.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T17:04:50.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T17:04:50.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T17:04:50.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:04:50.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T17:04:50.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:04:50.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:04:50.610 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:04:50.610 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:50.610 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:04:50.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:04:50.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:04:50.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:04:50.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:04:50.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:04:50.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:04:50.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:04:50.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:04:50.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:04:50.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T17:04:50.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T17:04:50.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T17:04:50.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T17:04:50.612 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-09-17T17:04:50.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:04:50.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-17T17:04:50.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T17:04:50.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T17:04:50.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T17:04:50.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T17:04:50.640 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:04:50.638+0000 7fc7825c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:04:50.642 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:04:50.640+0000 7fc7825c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:04:50.643 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:04:50.642+0000 7fc7825c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:04:51.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T17:04:51.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:04:51.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T17:04:51.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:04:51.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:04:51.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:51.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:04:51.039 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T17:04:51.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:51.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:04:51.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:51.744 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:04:51.742+0000 7fc7825c5640 -1 Falling back to public interface 2024-09-17T17:04:52.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:52.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:52.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:04:52.474 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:04:52.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:52.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:04:52.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:53.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:53.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:53.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:04:53.909 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T17:04:53.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:53.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:04:54.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:04:54.540+0000 7fc7825c5640 -1 osd.1 39 log_to_monitors true 2024-09-17T17:04:55.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:04:55.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:55.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:04:55.309 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T17:04:55.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:55.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:04:55.949 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 40 last_clean_interval [14,39) [v2:127.0.0.1:6810/2719028710,v1:127.0.0.1:6811/2719028710] [v2:127.0.0.1:6812/2719028710,v1:127.0.0.1:6813/2719028710] exists,up efc4e443-585d-40ad-98d5-637384df5871 2024-09-17T17:04:55.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:04:55.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:04:55.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:04:55.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T17:04:55.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T17:04:55.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T17:04:55.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T17:04:55.950 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T17:04:55.950 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T17:04:55.950 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T17:04:55.950 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T17:04:55.950 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T17:04:55.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T17:04:56.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T17:04:56.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T17:04:56.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T17:04:56.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T17:04:56.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T17:04:56.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T17:04:56.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T17:04:56.449 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:04:56.449 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T17:04:56.450 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T17:04:56.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T17:04:56.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:04:56.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T17:04:56.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803813 2024-09-17T17:04:56.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803813 2024-09-17T17:04:56.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803813' 2024-09-17T17:04:56.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:04:56.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T17:04:56.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=188978561027 2024-09-17T17:04:56.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 188978561027 2024-09-17T17:04:56.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803813 1-188978561027' 2024-09-17T17:04:56.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:04:56.698 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T17:04:56.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345944 2024-09-17T17:04:56.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345944 2024-09-17T17:04:56.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803813 1-188978561027 2-85899345944' 2024-09-17T17:04:56.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:04:56.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T17:04:56.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018890 2024-09-17T17:04:56.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018890 2024-09-17T17:04:56.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803813 1-188978561027 2-85899345944 3-128849018890' 2024-09-17T17:04:56.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:04:56.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803813 2024-09-17T17:04:56.926 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:04:56.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T17:04:56.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803813 2024-09-17T17:04:56.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:04:56.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803813 2024-09-17T17:04:56.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-09-17T17:04:56.928 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803813 2024-09-17T17:04:56.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:04:57.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803811 -lt 25769803813 2024-09-17T17:04:57.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T17:04:58.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T17:04:58.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:04:58.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803813 -lt 25769803813 2024-09-17T17:04:58.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:04:58.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-188978561027 2024-09-17T17:04:58.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:04:58.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T17:04:58.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-188978561027 2024-09-17T17:04:58.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:04:58.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=188978561027 2024-09-17T17:04:58.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 188978561027' 2024-09-17T17:04:58.790 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 188978561027 2024-09-17T17:04:58.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T17:04:59.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 188978561027 -lt 188978561027 2024-09-17T17:04:59.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:04:59.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345944 2024-09-17T17:04:59.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:04:59.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T17:04:59.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345944 2024-09-17T17:04:59.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:04:59.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345944 2024-09-17T17:04:59.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345944' 2024-09-17T17:04:59.208 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 85899345944 2024-09-17T17:04:59.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T17:04:59.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345944 -lt 85899345944 2024-09-17T17:04:59.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:04:59.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018890 2024-09-17T17:04:59.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:04:59.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T17:04:59.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018890 2024-09-17T17:04:59.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:04:59.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018890 2024-09-17T17:04:59.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018890' 2024-09-17T17:04:59.643 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 128849018890 2024-09-17T17:04:59.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T17:05:00.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018891 -lt 128849018890 2024-09-17T17:05:00.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T17:05:00.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:05:00.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:05:00.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T17:05:00.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T17:05:00.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T17:05:00.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T17:05:00.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T17:05:00.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T17:05:00.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T17:05:00.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T17:05:01.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T17:05:01.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T17:05:01.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:05:01.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:05:01.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T17:05:01.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T17:05:01.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T17:05:01.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-87940-1 2024-09-17T17:05:01.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T17:05:01.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T17:05:01.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-87940-1 2024-09-17T17:05:01.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T17:05:01.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T17:05:01.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-remove-87940-1 td/test-erasure-eio/COPY 2024-09-17T17:05:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T17:05:01.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T17:05:01.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-17T17:05:01.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2024-09-17T17:05:01.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: local another_shard_id=2 2024-09-17T17:05:01.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-87940-1 td/test-erasure-eio 2 2024-09-17T17:05:01.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2024-09-17T17:05:01.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2024-09-17T17:05:01.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2024-09-17T17:05:01.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2024-09-17T17:05:01.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2024-09-17T17:05:01.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2024-09-17T17:05:01.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-87940-1 2024-09-17T17:05:01.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2024-09-17T17:05:01.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2024-09-17T17:05:01.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2024-09-17T17:05:01.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=2 2024-09-17T17:05:01.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2024-09-17T17:05:01.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-87940-1 2024-09-17T17:05:01.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T17:05:01.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-remove-87940-1 2024-09-17T17:05:01.610 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-remove-87940-1 2024-09-17T17:05:01.610 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T17:05:02.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T17:05:02.031 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:05:02.032 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T17:05:02.032 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T17:05:02.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2024-09-17T17:05:02.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2024-09-17T17:05:02.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=2 2024-09-17T17:05:02.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 2 obj-remove-87940-1 remove 2024-09-17T17:05:02.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-17T17:05:02.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T17:05:02.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-17T17:05:02.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T17:05:02.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-remove-87940-1 remove 2024-09-17T17:05:02.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-17T17:05:02.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T17:05:02.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-17T17:05:02.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T17:05:02.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-17T17:05:02.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:05:02.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:05:02.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:05:02.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:05:02.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:05:02.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:05:02.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-remove-87940-1 remove 2024-09-17T17:05:02.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-17T17:05:02.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T17:05:02.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-17T17:05:02.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T17:05:02.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-17T17:05:02.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-remove-87940-1 remove 2024-09-17T17:05:04.457 INFO:tasks.workunit.client.0.smithi162.stdout:remove 2#2:7040a0dd:::obj-remove-87940-1:head# 2024-09-17T17:05:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-17T17:05:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T17:05:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T17:05:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T17:05:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T17:05:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T17:05:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:05:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:05:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:05:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:05:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T17:05:04.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T17:05:04.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T17:05:04.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T17:05:04.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:05:04.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T17:05:04.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:05:04.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:05:04.975 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:05:04.975 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:05:04.975 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:05:04.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:05:04.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:05:04.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:05:04.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:05:04.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:05:04.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:05:04.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:05:04.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:05:04.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:05:04.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T17:05:04.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T17:05:04.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T17:05:04.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T17:05:04.977 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-09-17T17:05:04.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:05:04.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-17T17:05:04.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T17:05:04.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T17:05:04.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T17:05:04.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T17:05:05.005 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:05.003+0000 7fc88520e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:05.007 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:05.005+0000 7fc88520e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:05.008 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:05.006+0000 7fc88520e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:05.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T17:05:05.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:05:05.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T17:05:05.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:05:05.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:05:05.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:05.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:05:05.412 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-09-17T17:05:05.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:05.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:05:05.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:06.352 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:06.350+0000 7fc88520e640 -1 Falling back to public interface 2024-09-17T17:05:06.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:06.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:06.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:05:06.823 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:05:06.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:06.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:05:07.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:08.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:08.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:08.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:05:08.257 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T17:05:08.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:08.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:05:08.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:09.198 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:09.195+0000 7fc88520e640 -1 osd.2 45 log_to_monitors true 2024-09-17T17:05:09.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:09.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:09.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:05:09.677 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-09-17T17:05:09.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:09.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:05:10.175 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 50 up_thru 40 down_at 46 last_clean_interval [20,45) [v2:127.0.0.1:6818/1271862023,v1:127.0.0.1:6819/1271862023] [v2:127.0.0.1:6820/1271862023,v1:127.0.0.1:6821/1271862023] exists,up f7f97bae-fbc9-464d-802a-1b6f3e99546e 2024-09-17T17:05:10.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:05:10.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:05:10.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:05:10.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T17:05:10.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T17:05:10.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T17:05:10.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T17:05:10.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T17:05:10.176 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T17:05:10.176 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T17:05:10.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T17:05:10.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T17:05:10.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T17:05:10.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T17:05:10.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T17:05:10.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T17:05:10.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T17:05:10.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T17:05:10.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T17:05:10.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T17:05:10.685 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:05:10.685 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T17:05:10.685 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T17:05:10.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T17:05:10.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:05:10.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T17:05:10.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803818 2024-09-17T17:05:10.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803818 2024-09-17T17:05:10.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803818' 2024-09-17T17:05:10.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:05:10.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T17:05:11.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=188978561032 2024-09-17T17:05:11.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 188978561032 2024-09-17T17:05:11.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803818 1-188978561032' 2024-09-17T17:05:11.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:05:11.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T17:05:11.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364803 2024-09-17T17:05:11.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364803 2024-09-17T17:05:11.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803818 1-188978561032 2-214748364803' 2024-09-17T17:05:11.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:05:11.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T17:05:11.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018895 2024-09-17T17:05:11.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018895 2024-09-17T17:05:11.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803818 1-188978561032 2-214748364803 3-128849018895' 2024-09-17T17:05:11.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:05:11.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803818 2024-09-17T17:05:11.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:05:11.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T17:05:11.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803818 2024-09-17T17:05:11.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:05:11.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803818 2024-09-17T17:05:11.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803818' 2024-09-17T17:05:11.234 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 25769803818 2024-09-17T17:05:11.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:05:11.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803816 -lt 25769803818 2024-09-17T17:05:11.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T17:05:12.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T17:05:12.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:05:13.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803818 -lt 25769803818 2024-09-17T17:05:13.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:05:13.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-188978561032 2024-09-17T17:05:13.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:05:13.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T17:05:13.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-188978561032 2024-09-17T17:05:13.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:05:13.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=188978561032 2024-09-17T17:05:13.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 188978561032' 2024-09-17T17:05:13.051 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 188978561032 2024-09-17T17:05:13.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T17:05:13.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 188978561032 -lt 188978561032 2024-09-17T17:05:13.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:05:13.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-214748364803 2024-09-17T17:05:13.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:05:13.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T17:05:13.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-214748364803 2024-09-17T17:05:13.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:05:13.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364803 2024-09-17T17:05:13.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-09-17T17:05:13.479 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 214748364803 2024-09-17T17:05:13.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T17:05:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364803 -lt 214748364803 2024-09-17T17:05:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:05:13.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018895 2024-09-17T17:05:13.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:05:13.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T17:05:13.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018895 2024-09-17T17:05:13.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:05:13.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018895 2024-09-17T17:05:13.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018895' 2024-09-17T17:05:13.871 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 128849018895 2024-09-17T17:05:13.872 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T17:05:14.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018896 -lt 128849018895 2024-09-17T17:05:14.338 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T17:05:14.338 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:05:14.338 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:05:14.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T17:05:14.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T17:05:14.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T17:05:14.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T17:05:14.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T17:05:14.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T17:05:14.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T17:05:14.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T17:05:15.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T17:05:15.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T17:05:15.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:05:15.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:05:15.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T17:05:15.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T17:05:15.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T17:05:15.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: '[' 1 -eq 1 -a 2 -eq 2 ']' 2024-09-17T17:05:15.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:188: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-87940-1 2024-09-17T17:05:15.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T17:05:15.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T17:05:15.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-87940-1 2024-09-17T17:05:15.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T17:05:15.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T17:05:15.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-remove-87940-1 td/test-erasure-eio/COPY 2024-09-17T17:05:15.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T17:05:15.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T17:05:15.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-17T17:05:15.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:310: TEST_rados_get_subread_missing: delete_erasure_coded_pool pool-jerasure 2024-09-17T17:05:15.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T17:05:15.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-17T17:05:16.360 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-17T17:05:16.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-17T17:05:17.057 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-17T17:05:17.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-17T17:05:17.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T17:05:17.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:05:17.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T17:05:17.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:05:17.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:05:17.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:05:17.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:05:17.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:05:17.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:05:17.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:05:17.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:05:17.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:05:17.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:05:17.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:05:17.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:05:17.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:05:17.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:05:17.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:05:17.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:05:17.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:05:17.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:05:17.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T17:05:17.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:05:17.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:05:17.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:05:17.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T17:05:17.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:05:17.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:05:17.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-17T17:05:17.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-17T17:05:17.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-17T17:05:17.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-17T17:05:17.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T17:05:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:05:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T17:05:17.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:05:17.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:05:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:05:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:05:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:05:17.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:05:17.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:05:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:05:17.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:05:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:05:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:05:17.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:05:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:05:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:05:17.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:05:17.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:05:17.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:05:17.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:05:17.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T17:05:17.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:05:17.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:05:17.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:05:17.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T17:05:17.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:05:17.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:05:17.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-17T17:05:17.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T17:05:17.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:05:17.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:05:17.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.87940 2024-09-17T17:05:17.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T17:05:17.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T17:05:17.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T17:05:17.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-17T17:05:17.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-17T17:05:17.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-17T17:05:17.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T17:05:17.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T17:05:17.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T17:05:17.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-17T17:05:17.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-17T17:05:17.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T17:05:17.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:05:17.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:05:17.683 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:05:17.683 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:05:17.683 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:05:17.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:05:17.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T17:05:17.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T17:05:17.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T17:05:17.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:05:17.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T17:05:17.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T17:05:17.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T17:05:17.860 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T17:05:17.860 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T17:05:17.861 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T17:05:17.861 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:05:17.861 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:05:17.861 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:05:17.861 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T17:05:17.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:05:17.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get fsid 2024-09-17T17:05:17.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T17:05:17.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:05:17.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T17:05:17.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T17:05:17.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T17:05:17.939 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T17:05:17.939 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T17:05:17.939 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T17:05:17.939 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:05:17.939 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:05:17.939 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:05:17.939 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T17:05:17.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:05:17.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get mon_host 2024-09-17T17:05:18.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-17T17:05:18.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-17T17:05:18.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T17:05:18.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T17:05:18.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T17:05:18.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-17T17:05:18.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T17:05:18.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T17:05:18.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:05:18.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:05:18.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:05:18.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:05:18.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:05:18.866 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:05:18.866 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:05:18.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:05:18.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-17T17:05:18.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T17:05:19.175 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T17:05:19.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T17:05:20.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-17T17:05:20.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T17:05:20.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T17:05:20.187 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:05:20.187 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:05:20.187 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:05:20.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T17:05:20.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-17T17:05:20.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-mon.a.asok log flush 2024-09-17T17:05:20.262 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T17:05:20.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-17T17:05:20.271 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T17:05:17.712+0000 7fce7371ec80 0 load: jerasure load: lrc 2024-09-17T17:05:20.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_0 td/test-erasure-eio 2024-09-17T17:05:20.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:350: TEST_rados_get_with_subreadall_eio_shard_0: local dir=td/test-erasure-eio 2024-09-17T17:05:20.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:351: TEST_rados_get_with_subreadall_eio_shard_0: local shard_id=0 2024-09-17T17:05:20.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:353: TEST_rados_get_with_subreadall_eio_shard_0: setup_osds 4 2024-09-17T17:05:20.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-17T17:05:20.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-17T17:05:20.272 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-17T17:05:20.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-17T17:05:20.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T17:05:20.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-17T17:05:20.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T17:05:20.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:05:20.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T17:05:20.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:05:20.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-17T17:05:20.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:05:20.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:05:20.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:05:20.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:05:20.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-17T17:05:20.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-17T17:05:20.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:05:20.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:05:20.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:05:20.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:05:20.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:05:20.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:05:20.282 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:05:20.282 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:05:20.282 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:05:20.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:05:20.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:05:20.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:05:20.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:05:20.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:05:20.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:05:20.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:05:20.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:05:20.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:05:20.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:05:20.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:05:20.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:05:20.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:05:20.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-17T17:05:20.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:05:20.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a73e2baa-4b7d-45fb-b2e5-8b4fd78ee133 2024-09-17T17:05:20.284 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 a73e2baa-4b7d-45fb-b2e5-8b4fd78ee133 2024-09-17T17:05:20.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a73e2baa-4b7d-45fb-b2e5-8b4fd78ee133' 2024-09-17T17:05:20.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:05:20.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDQtulmLzaDERAA/S+C83PGxeUrCrX60cJDcg== 2024-09-17T17:05:20.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDQtulmLzaDERAA/S+C83PGxeUrCrX60cJDcg=="}' 2024-09-17T17:05:20.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a73e2baa-4b7d-45fb-b2e5-8b4fd78ee133 -i td/test-erasure-eio/0/new.json 2024-09-17T17:05:20.551 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:05:20.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-17T17:05:20.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQtulmLzaDERAA/S+C83PGxeUrCrX60cJDcg== --osd-uuid a73e2baa-4b7d-45fb-b2e5-8b4fd78ee133 2024-09-17T17:05:20.591 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:20.588+0000 7f7387f4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:20.620 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:20.617+0000 7f7387f4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:20.621 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:20.618+0000 7f7387f4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:20.621 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:20.618+0000 7f7387f4d640 -1 bdev(0x555fc78f9c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-17T17:05:20.621 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:20.618+0000 7f7387f4d640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-17T17:05:26.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-17T17:05:26.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:05:26.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T17:05:26.843 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T17:05:26.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:05:27.382 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T17:05:27.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T17:05:27.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:05:27.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:05:27.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:05:27.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:05:27.409 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:27.407+0000 7faac1bc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:27.442 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:27.440+0000 7faac1bc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:27.443 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:27.441+0000 7faac1bc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:27.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T17:05:27.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:05:27.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T17:05:27.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:05:27.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:05:27.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:27.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:05:27.767 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:05:27.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:27.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:28.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:28.774 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:28.772+0000 7faac1bc5640 -1 Falling back to public interface 2024-09-17T17:05:29.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:29.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:29.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:05:29.183 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:05:29.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:29.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:29.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:30.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:30.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:30.618 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:05:30.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:05:30.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:30.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:31.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:31.587 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:31.585+0000 7faac1bc5640 -1 osd.0 0 log_to_monitors true 2024-09-17T17:05:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:32.024 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:05:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:05:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:32.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:33.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:33.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:33.546 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:05:33.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:05:33.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:33.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:33.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:34.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:34.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:34.960 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T17:05:34.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:05:34.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:34.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:35.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:36.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:36.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:36.496 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T17:05:36.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:05:36.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:36.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:36.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:37.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:37.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:37.902 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T17:05:37.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:05:37.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:37.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:38.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:39.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:39.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:39.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:05:39.379 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T17:05:39.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:39.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:39.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:40.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:40.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:40.797 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T17:05:40.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:05:40.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:40.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:41.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:42.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:42.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:42.340 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T17:05:42.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:05:42.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:42.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:42.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:43.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:43.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:43.751 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:05:43.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:05:43.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:43.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:44.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:45.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:45.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:45.477 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:05:45.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:05:45.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:45.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:46.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:47.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:47.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:47.535 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:05:47.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:05:47.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:47.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:48.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:49.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:49.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:49.180 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:05:49.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:05:49.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:49.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:49.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:50.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:50.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:50.813 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:05:50.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:05:50.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:50.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:51.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:52.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:52.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:52.264 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:05:52.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:05:52.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:52.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:53.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:54.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:54.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:54.001 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:05:54.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:05:54.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:54.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:54.596 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3424439546,v1:127.0.0.1:6803/3424439546] [v2:127.0.0.1:6804/3424439546,v1:127.0.0.1:6805/3424439546] exists,up a73e2baa-4b7d-45fb-b2e5-8b4fd78ee133 2024-09-17T17:05:54.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:05:54.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:05:54.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:05:54.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T17:05:54.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-17T17:05:54.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T17:05:54.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:05:54.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T17:05:54.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:05:54.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T17:05:54.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:05:54.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:05:54.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:05:54.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:05:54.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T17:05:54.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T17:05:54.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:05:54.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:05:54.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:05:54.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:05:54.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:05:54.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:05:54.601 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:05:54.601 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:05:54.601 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:05:54.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:05:54.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:05:54.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:05:54.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:05:54.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:05:54.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:05:54.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:05:54.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:05:54.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:05:54.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:05:54.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:05:54.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:05:54.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:05:54.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T17:05:54.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:05:54.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=27151705-d92f-45e4-a29b-a924091f2fcb 2024-09-17T17:05:54.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 27151705-d92f-45e4-a29b-a924091f2fcb' 2024-09-17T17:05:54.603 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 27151705-d92f-45e4-a29b-a924091f2fcb 2024-09-17T17:05:54.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:05:54.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDytulmseW7JBAAhY17vj24E+u2UXHdA/5grw== 2024-09-17T17:05:54.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDytulmseW7JBAAhY17vj24E+u2UXHdA/5grw=="}' 2024-09-17T17:05:54.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 27151705-d92f-45e4-a29b-a924091f2fcb -i td/test-erasure-eio/1/new.json 2024-09-17T17:05:55.399 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:05:55.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-17T17:05:55.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDytulmseW7JBAAhY17vj24E+u2UXHdA/5grw== --osd-uuid 27151705-d92f-45e4-a29b-a924091f2fcb 2024-09-17T17:05:55.440 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:55.438+0000 7fbc30a0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:55.441 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:55.439+0000 7fbc30a0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:55.442 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:55.440+0000 7fbc30a0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:55.443 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:55.441+0000 7fbc30a0b640 -1 bdev(0x56530031bc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-17T17:05:55.443 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:05:55.441+0000 7fbc30a0b640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-17T17:06:01.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-17T17:06:01.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:06:01.977 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T17:06:01.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T17:06:01.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:06:02.519 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T17:06:02.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T17:06:02.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:06:02.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:06:02.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:06:02.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:06:02.546 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:06:02.543+0000 7f85f2a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:02.563 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:06:02.560+0000 7f85f2a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:02.564 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:06:02.562+0000 7f85f2a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:02.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T17:06:02.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:06:02.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T17:06:02.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:06:02.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:06:02.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:02.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:06:02.979 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:06:02.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:02.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:03.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:04.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:06:04.162+0000 7f85f2a0f640 -1 Falling back to public interface 2024-09-17T17:06:04.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:04.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:04.405 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:06:04.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:06:04.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:04.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:04.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:05.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:05.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:05.832 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:06:05.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:06:05.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:05.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:06.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:07.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:07.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:07.267 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:06:07.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:06:07.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:07.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:07.360 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:06:07.356+0000 7f85f2a0f640 -1 osd.1 0 log_to_monitors true 2024-09-17T17:06:07.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:08.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:08.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:08.679 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:06:08.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:06:08.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:08.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:09.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:10.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:10.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:10.312 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T17:06:10.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:06:10.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:10.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:10.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:11.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:11.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:11.709 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T17:06:11.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:06:11.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:11.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:12.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:13.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:13.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:13.166 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T17:06:13.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:06:13.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:13.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:13.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:14.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:14.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:14.604 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T17:06:14.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:06:14.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:14.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:14.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:15.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:15.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:15.995 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T17:06:15.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:06:15.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:15.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:16.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:17.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:17.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:17.550 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T17:06:17.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:06:17.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:17.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:17.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:18.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:18.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:18.950 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:06:18.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:06:18.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:18.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:19.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:20.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:20.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:20.369 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:06:20.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:06:20.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:20.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:20.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:21.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:21.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:21.931 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:06:21.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:06:21.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:21.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:22.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:23.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:23.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:23.689 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:06:23.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:06:23.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:23.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:24.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:25.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:25.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:25.685 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:06:25.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:06:25.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:25.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:26.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:27.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:27.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:27.431 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:06:27.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:06:27.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:27.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:28.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:29.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:29.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:29.174 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:06:29.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:06:29.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:29.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:30.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:31.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:31.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:31.009 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T17:06:31.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T17:06:31.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:31.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:31.452 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2048589909,v1:127.0.0.1:6811/2048589909] [v2:127.0.0.1:6812/2048589909,v1:127.0.0.1:6813/2048589909] exists,up 27151705-d92f-45e4-a29b-a924091f2fcb 2024-09-17T17:06:31.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:06:31.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:06:31.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:06:31.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T17:06:31.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-17T17:06:31.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T17:06:31.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:06:31.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T17:06:31.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:06:31.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T17:06:31.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:06:31.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:06:31.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:06:31.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:06:31.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T17:06:31.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T17:06:31.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:06:31.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:06:31.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:06:31.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:06:31.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:06:31.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:06:31.457 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:06:31.457 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:06:31.457 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:06:31.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:06:31.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:06:31.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:06:31.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:06:31.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:06:31.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:06:31.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:06:31.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:06:31.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:06:31.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:06:31.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:06:31.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:06:31.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:06:31.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T17:06:31.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:06:31.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fffdb37d-ac44-49ff-9b71-1becabaecadf 2024-09-17T17:06:31.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 fffdb37d-ac44-49ff-9b71-1becabaecadf' 2024-09-17T17:06:31.459 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 fffdb37d-ac44-49ff-9b71-1becabaecadf 2024-09-17T17:06:31.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:06:31.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAXt+lmL6coHBAAwvZEU7ul0h93M6BpWyaVyA== 2024-09-17T17:06:31.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAXt+lmL6coHBAAwvZEU7ul0h93M6BpWyaVyA=="}' 2024-09-17T17:06:31.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fffdb37d-ac44-49ff-9b71-1becabaecadf -i td/test-erasure-eio/2/new.json 2024-09-17T17:06:31.987 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:06:31.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-17T17:06:31.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAXt+lmL6coHBAAwvZEU7ul0h93M6BpWyaVyA== --osd-uuid fffdb37d-ac44-49ff-9b71-1becabaecadf 2024-09-17T17:06:32.028 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:06:32.026+0000 7f985f019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:32.125 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:06:32.123+0000 7f985f019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:32.126 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:06:32.124+0000 7f985f019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:32.126 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:06:32.125+0000 7f985f019640 -1 bdev(0x563c382b9c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-17T17:06:32.126 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:06:32.125+0000 7f985f019640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-17T17:06:38.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-17T17:06:38.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:06:38.446 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T17:06:38.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T17:06:38.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:06:38.979 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T17:06:38.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T17:06:38.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:06:38.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:06:38.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:06:38.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:06:39.007 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:06:39.005+0000 7fe7ab3b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:39.029 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:06:39.028+0000 7fe7ab3b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:39.031 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:06:39.029+0000 7fe7ab3b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:39.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T17:06:39.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:06:39.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T17:06:39.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:06:39.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:06:39.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:39.429 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:06:39.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:06:39.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:39.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:39.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:40.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:40.846 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:06:40.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:40.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:06:40.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:40.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:40.869 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:06:40.867+0000 7fe7ab3b2640 -1 Falling back to public interface 2024-09-17T17:06:41.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:42.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:42.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:42.248 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:06:42.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:06:42.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:42.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:42.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:43.527 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:06:43.525+0000 7fe7ab3b2640 -1 osd.2 0 log_to_monitors true 2024-09-17T17:06:43.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:43.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:43.683 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:06:43.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:06:43.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:43.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:44.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:45.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:45.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:45.212 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:06:45.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:06:45.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:45.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:45.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:46.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:46.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:46.670 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T17:06:46.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:06:46.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:46.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:47.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:48.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:48.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:48.072 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T17:06:48.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:06:48.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:48.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:48.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:49.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:49.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:49.584 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T17:06:49.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:06:49.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:49.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:50.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:51.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:51.019 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T17:06:51.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:51.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:06:51.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:51.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:51.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:52.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:52.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:52.511 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T17:06:52.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:06:52.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:52.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:52.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:53.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:53.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:53.920 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T17:06:53.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:06:53.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:53.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:54.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:55.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:55.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:55.368 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:06:55.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:06:55.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:55.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:56.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:57.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:57.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:57.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:06:57.029 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:06:57.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:57.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:57.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:58.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:58.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:58.532 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:06:58.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:06:58.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:58.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:59.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:00.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:00.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:00.269 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:07:00.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:07:00.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:00.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:07:01.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:02.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:02.021 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:07:02.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:02.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:07:02.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:02.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:07:02.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:03.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:03.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:03.857 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:07:03.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:07:03.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:03.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:07:04.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:05.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:05.575 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:07:05.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:05.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:07:05.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:05.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:07:06.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:07.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:07.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:07.489 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T17:07:07.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T17:07:07.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:07.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:07:08.195 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/23869949,v1:127.0.0.1:6819/23869949] [v2:127.0.0.1:6820/23869949,v1:127.0.0.1:6821/23869949] exists,up fffdb37d-ac44-49ff-9b71-1becabaecadf 2024-09-17T17:07:08.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:07:08.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:07:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:07:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T17:07:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-17T17:07:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T17:07:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:07:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T17:07:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:07:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T17:07:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:07:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:07:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:07:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:07:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T17:07:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T17:07:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:07:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:07:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:07:08.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:07:08.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:07:08.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:07:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:07:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:07:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:07:08.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:07:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:07:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:07:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:07:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:07:08.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:07:08.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:07:08.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:07:08.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:07:08.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:07:08.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:07:08.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:07:08.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:07:08.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T17:07:08.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:07:08.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=011b8994-294c-474b-9376-5ab38e526f22 2024-09-17T17:07:08.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 011b8994-294c-474b-9376-5ab38e526f22' 2024-09-17T17:07:08.203 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 011b8994-294c-474b-9376-5ab38e526f22 2024-09-17T17:07:08.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:07:08.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA8t+lmIOviDBAASmIXxjfx0I2COz68vz4l5w== 2024-09-17T17:07:08.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA8t+lmIOviDBAASmIXxjfx0I2COz68vz4l5w=="}' 2024-09-17T17:07:08.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 011b8994-294c-474b-9376-5ab38e526f22 -i td/test-erasure-eio/3/new.json 2024-09-17T17:07:08.990 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:07:09.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-17T17:07:09.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8t+lmIOviDBAASmIXxjfx0I2COz68vz4l5w== --osd-uuid 011b8994-294c-474b-9376-5ab38e526f22 2024-09-17T17:07:09.031 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:07:09.029+0000 7f892b60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:07:09.124 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:07:09.122+0000 7f892b60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:07:09.125 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:07:09.123+0000 7f892b60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:07:09.126 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:07:09.124+0000 7f892b60b640 -1 bdev(0x55fdfa237c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-17T17:07:09.126 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:07:09.124+0000 7f892b60b640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-17T17:07:15.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-17T17:07:15.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:07:15.168 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-17T17:07:15.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T17:07:15.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:07:15.700 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T17:07:15.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T17:07:15.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:07:15.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:07:15.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:07:15.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:07:15.728 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:07:15.726+0000 7efd6d814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:07:15.747 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:07:15.744+0000 7efd6d814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:07:15.748 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:07:15.746+0000 7efd6d814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:07:16.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T17:07:16.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:07:16.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T17:07:16.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:07:16.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:07:16.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:16.122 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:07:16.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:07:16.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:16.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:16.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:17.358 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:07:17.356+0000 7efd6d814640 -1 Falling back to public interface 2024-09-17T17:07:17.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:17.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:17.554 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:07:17.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:07:17.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:17.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:17.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:18.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:18.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:18.989 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:07:18.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:07:18.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:18.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:19.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:20.335 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:07:20.333+0000 7efd6d814640 -1 osd.3 0 log_to_monitors true 2024-09-17T17:07:20.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:20.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:20.415 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:07:20.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:07:20.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:20.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:20.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:21.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:21.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:21.932 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:07:21.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:07:21.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:21.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:22.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:23.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:23.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:23.540 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T17:07:23.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:07:23.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:23.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:23.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:24.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:24.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:24.946 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T17:07:24.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:07:24.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:24.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:25.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:26.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:26.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:26.485 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T17:07:26.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:07:26.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:26.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:26.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:27.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:27.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:27.907 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T17:07:27.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:07:27.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:27.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:28.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:29.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:29.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:29.363 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T17:07:29.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:07:29.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:29.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:29.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:30.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:30.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:30.776 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T17:07:30.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:07:30.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:30.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:31.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:32.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:32.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:32.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:07:32.301 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:07:32.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:32.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:33.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:33.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:33.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:07:33.698 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:07:33.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:33.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:34.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:35.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:35.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:35.535 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:07:35.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:07:35.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:35.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:36.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:37.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:37.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:37.596 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:07:37.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:07:37.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:37.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:38.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:39.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:39.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:39.130 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:07:39.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:07:39.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:39.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:39.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:40.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:40.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:40.592 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:07:40.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:07:40.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:40.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:41.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:42.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:42.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:42.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:07:42.293 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:07:42.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:42.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:44.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:44.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:44.202 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T17:07:44.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T17:07:44.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:44.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:44.641 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3396484619,v1:127.0.0.1:6827/3396484619] [v2:127.0.0.1:6828/3396484619,v1:127.0.0.1:6829/3396484619] exists,up 011b8994-294c-474b-9376-5ab38e526f22 2024-09-17T17:07:44.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:07:44.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:07:44.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:07:44.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-17T17:07:44.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T17:07:44.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T17:07:44.643 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:07:44.643 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:07:44.643 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:07:44.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T17:07:44.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-17T17:07:44.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.0.asok log flush 2024-09-17T17:07:44.722 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T17:07:44.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-17T17:07:44.732 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T17:05:28.777+0000 7faac1bc5640 0 load: jerasure load: lrc 2024-09-17T17:07:44.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:355: TEST_rados_get_with_subreadall_eio_shard_0: local poolname=pool-jerasure 2024-09-17T17:07:44.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:356: TEST_rados_get_with_subreadall_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-17T17:07:44.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T17:07:44.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-17T17:07:44.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-17T17:07:44.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-17T17:07:44.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-17T17:07:44.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-17T17:07:44.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-17T17:07:45.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-17T17:07:45.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-17T17:07:47.338 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-17T17:07:47.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T17:07:48.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-17T17:07:48.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T17:07:48.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T17:07:48.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T17:07:48.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T17:07:48.354 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T17:07:48.355 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T17:07:48.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T17:07:48.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T17:07:48.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T17:07:48.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T17:07:48.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T17:07:48.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T17:07:48.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T17:07:48.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T17:07:48.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T17:07:48.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T17:07:48.843 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:07:48.843 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T17:07:48.843 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T17:07:48.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T17:07:48.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:07:48.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T17:07:48.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803809 2024-09-17T17:07:48.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803809 2024-09-17T17:07:48.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809' 2024-09-17T17:07:48.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:07:48.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T17:07:49.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574867 2024-09-17T17:07:49.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574867 2024-09-17T17:07:49.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-55834574867' 2024-09-17T17:07:49.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:07:49.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T17:07:49.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345938 2024-09-17T17:07:49.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345938 2024-09-17T17:07:49.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-55834574867 2-85899345938' 2024-09-17T17:07:49.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:07:49.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T17:07:49.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018889 2024-09-17T17:07:49.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018889 2024-09-17T17:07:49.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-55834574867 2-85899345938 3-128849018889' 2024-09-17T17:07:49.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:07:49.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803809 2024-09-17T17:07:49.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:07:49.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T17:07:49.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803809 2024-09-17T17:07:49.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:07:49.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803809 2024-09-17T17:07:49.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2024-09-17T17:07:49.307 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803809 2024-09-17T17:07:49.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:07:49.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803808 -lt 25769803809 2024-09-17T17:07:49.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T17:07:50.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T17:07:50.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:07:51.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803810 -lt 25769803809 2024-09-17T17:07:51.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:07:51.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574867 2024-09-17T17:07:51.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:07:51.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T17:07:51.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574867 2024-09-17T17:07:51.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:07:51.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574867 2024-09-17T17:07:51.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574867' 2024-09-17T17:07:51.178 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574867 2024-09-17T17:07:51.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T17:07:51.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574868 -lt 55834574867 2024-09-17T17:07:51.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:07:51.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345938 2024-09-17T17:07:51.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:07:51.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T17:07:51.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345938 2024-09-17T17:07:51.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:07:51.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345938 2024-09-17T17:07:51.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345938' 2024-09-17T17:07:51.568 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 85899345938 2024-09-17T17:07:51.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T17:07:51.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345938 -lt 85899345938 2024-09-17T17:07:51.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:07:51.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018889 2024-09-17T17:07:51.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:07:51.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T17:07:51.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018889 2024-09-17T17:07:51.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:07:51.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018889 2024-09-17T17:07:51.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018889' 2024-09-17T17:07:51.957 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 128849018889 2024-09-17T17:07:51.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T17:07:52.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018890 -lt 128849018889 2024-09-17T17:07:52.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T17:07:52.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:07:52.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:07:52.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T17:07:52.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T17:07:52.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T17:07:52.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T17:07:52.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T17:07:52.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T17:07:52.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T17:07:52.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T17:07:53.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T17:07:53.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T17:07:53.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:07:53.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:07:53.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T17:07:53.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T17:07:53.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T17:07:53.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:358: TEST_rados_get_with_subreadall_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 recovery 2024-09-17T17:07:53.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-17T17:07:53.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-17T17:07:53.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-17T17:07:53.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-17T17:07:53.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2024-09-17T17:07:53.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-17T17:07:53.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2024-09-17T17:07:53.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-17T17:07:53.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-87940-0 2024-09-17T17:07:53.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-87940-0 2024-09-17T17:07:53.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T17:07:53.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T17:07:53.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-87940-0 2024-09-17T17:07:53.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:07:53.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T17:07:53.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:07:53.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T17:07:53.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:07:53.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T17:07:53.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:07:53.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T17:07:53.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-87940-0 td/test-erasure-eio/ORIGINAL 2024-09-17T17:07:53.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-87940-0 td/test-erasure-eio 0 2024-09-17T17:07:53.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T17:07:53.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T17:07:53.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T17:07:53.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T17:07:53.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T17:07:53.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T17:07:53.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-87940-0 2024-09-17T17:07:53.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T17:07:53.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T17:07:53.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T17:07:53.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-17T17:07:53.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T17:07:53.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-87940-0 2024-09-17T17:07:53.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T17:07:53.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-87940-0 2024-09-17T17:07:53.974 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-87940-0 2024-09-17T17:07:53.974 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T17:07:54.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T17:07:54.403 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:07:54.403 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T17:07:54.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T17:07:54.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-17T17:07:54.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T17:07:54.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-17T17:07:54.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T17:07:54.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-17T17:07:54.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T17:07:54.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-17T17:07:54.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T17:07:54.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-17T17:07:54.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T17:07:54.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T17:07:54.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T17:07:54.406 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-17T17:07:54.406 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-17T17:07:54.407 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-17T17:07:54.407 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:07:54.407 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:07:54.407 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:07:54.407 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.3.asok 2024-09-17T17:07:54.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-17T17:07:54.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T17:07:54.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T17:07:54.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T17:07:54.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-17T17:07:54.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-17T17:07:54.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-17T17:07:54.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:07:54.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:07:54.494 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:07:54.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.3.asok 2024-09-17T17:07:54.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T17:07:54.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-87940-0 0 2024-09-17T17:07:54.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-87940-0 2024-09-17T17:07:54.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T17:07:54.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T17:07:54.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-87940-0 2024-09-17T17:07:54.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T17:07:54.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T17:07:54.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-87940-0 td/test-erasure-eio/COPY 2024-09-17T17:07:54.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T17:07:54.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T17:07:54.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2024-09-17T17:07:54.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-87940-0 2024-09-17T17:07:54.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T17:07:54.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-87940-0 2024-09-17T17:07:54.617 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-87940-0 2024-09-17T17:07:54.617 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T17:07:55.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T17:07:55.049 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:07:55.049 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T17:07:55.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T17:07:55.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2024-09-17T17:07:55.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2024-09-17T17:07:55.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2024-09-17T17:07:55.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-17T17:07:55.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:07:55.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:07:55.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:07:55.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:07:55.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:07:55.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:07:55.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2024-09-17T17:07:56.421 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already out. 2024-09-17T17:07:56.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-87940-0 2024-09-17T17:07:56.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2024-09-17T17:07:56.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T17:07:56.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-87940-0 2024-09-17T17:07:56.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-87940-0 2024-09-17T17:07:56.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T17:07:57.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T17:07:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:07:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-09-17T17:07:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 0 2024-09-17T17:07:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2024-09-17T17:07:58.011 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already in. 2024-09-17T17:07:58.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2024-09-17T17:07:58.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T17:07:58.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T17:07:58.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T17:07:58.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T17:07:58.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T17:07:58.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:07:58.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:07:58.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:07:58.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:07:58.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T17:07:58.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T17:07:58.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T17:07:58.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T17:07:58.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:07:58.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T17:07:58.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:07:58.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:07:58.027 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:07:58.027 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:07:58.027 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:07:58.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:07:58.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:07:58.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:07:58.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:07:58.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:07:58.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:07:58.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:07:58.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:07:58.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:07:58.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T17:07:58.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T17:07:58.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T17:07:58.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T17:07:58.028 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T17:07:58.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:07:58.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-17T17:07:58.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T17:07:58.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T17:07:58.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T17:07:58.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T17:07:58.052 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:07:58.050+0000 7fd6c2015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:07:58.054 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:07:58.052+0000 7fd6c2015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:07:58.055 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:07:58.053+0000 7fd6c2015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:07:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T17:07:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:07:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T17:07:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:07:58.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:07:58.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:58.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:07:58.541 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:07:58.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:58.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:07:59.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:59.672 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:07:59.670+0000 7fd6c2015640 -1 Falling back to public interface 2024-09-17T17:08:00.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:00.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:00.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:08:00.111 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:08:00.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:00.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:08:00.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:01.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:01.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:01.511 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:08:01.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:08:01.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:01.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:08:01.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:02.815 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:02.813+0000 7fd6c2015640 -1 osd.2 37 log_to_monitors true 2024-09-17T17:08:02.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:02.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:02.973 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:08:02.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:08:02.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:02.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:08:03.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:04.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:04.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:04.403 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:08:04.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:08:04.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:04.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:08:04.978 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 48 up_thru 22 down_at 38 last_clean_interval [20,37) [v2:127.0.0.1:6818/308751760,v1:127.0.0.1:6819/308751760] [v2:127.0.0.1:6820/308751760,v1:127.0.0.1:6821/308751760] exists,up fffdb37d-ac44-49ff-9b71-1becabaecadf 2024-09-17T17:08:04.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:08:04.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:08:04.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:08:04.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2024-09-17T17:08:04.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T17:08:04.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T17:08:04.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T17:08:04.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T17:08:04.980 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T17:08:04.980 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T17:08:04.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T17:08:04.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T17:08:04.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T17:08:05.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T17:08:05.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T17:08:05.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T17:08:05.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T17:08:05.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T17:08:05.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T17:08:05.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T17:08:05.517 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:08:05.517 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T17:08:05.517 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T17:08:05.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T17:08:05.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:08:05.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T17:08:05.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803815 2024-09-17T17:08:05.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803815 2024-09-17T17:08:05.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803815' 2024-09-17T17:08:05.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:08:05.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T17:08:05.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574873 2024-09-17T17:08:05.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574873 2024-09-17T17:08:05.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803815 1-55834574873' 2024-09-17T17:08:05.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:08:05.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T17:08:06.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=206158430211 2024-09-17T17:08:06.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 206158430211 2024-09-17T17:08:06.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803815 1-55834574873 2-206158430211' 2024-09-17T17:08:06.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:08:06.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T17:08:06.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018895 2024-09-17T17:08:06.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018895 2024-09-17T17:08:06.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803815 1-55834574873 2-206158430211 3-128849018895' 2024-09-17T17:08:06.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:08:06.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803815 2024-09-17T17:08:06.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:08:06.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T17:08:06.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803815 2024-09-17T17:08:06.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:08:06.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803815 2024-09-17T17:08:06.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803815' 2024-09-17T17:08:06.228 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803815 2024-09-17T17:08:06.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:08:06.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803812 -lt 25769803815 2024-09-17T17:08:06.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T17:08:07.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T17:08:07.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:08:08.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803815 -lt 25769803815 2024-09-17T17:08:08.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:08:08.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574873 2024-09-17T17:08:08.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:08:08.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T17:08:08.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574873 2024-09-17T17:08:08.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:08:08.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574873 2024-09-17T17:08:08.077 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574873 2024-09-17T17:08:08.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574873' 2024-09-17T17:08:08.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T17:08:08.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574873 -lt 55834574873 2024-09-17T17:08:08.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:08:08.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-206158430211 2024-09-17T17:08:08.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:08:08.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T17:08:08.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-206158430211 2024-09-17T17:08:08.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:08:08.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=206158430211 2024-09-17T17:08:08.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 206158430211' 2024-09-17T17:08:08.503 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 206158430211 2024-09-17T17:08:08.503 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T17:08:08.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 206158430211 -lt 206158430211 2024-09-17T17:08:08.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:08:08.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018895 2024-09-17T17:08:08.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:08:08.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T17:08:08.937 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018895 2024-09-17T17:08:08.937 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:08:08.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018895 2024-09-17T17:08:08.938 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 128849018895 2024-09-17T17:08:08.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018895' 2024-09-17T17:08:08.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T17:08:09.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018895 -lt 128849018895 2024-09-17T17:08:09.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T17:08:09.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:08:09.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:08:09.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T17:08:09.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T17:08:09.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T17:08:09.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T17:08:09.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T17:08:09.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T17:08:09.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T17:08:09.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T17:08:10.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T17:08:10.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T17:08:10.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:08:10.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:08:10.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T17:08:10.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T17:08:10.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T17:08:10.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:360: TEST_rados_get_with_subreadall_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-17T17:08:10.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T17:08:10.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-17T17:08:11.308 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-17T17:08:11.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-17T17:08:12.065 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-17T17:08:12.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-17T17:08:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T17:08:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:08:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T17:08:12.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:08:12.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:08:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:08:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:08:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:08:12.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:08:12.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:08:12.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:08:12.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:08:12.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:08:12.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:08:12.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:08:12.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:08:12.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:08:12.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:08:12.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:08:12.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:08:12.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:08:12.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T17:08:12.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:08:12.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:08:12.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:08:12.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T17:08:12.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:08:12.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:08:12.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-17T17:08:12.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-17T17:08:12.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-17T17:08:12.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-17T17:08:12.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T17:08:12.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:08:12.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T17:08:12.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:08:12.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:08:12.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:08:12.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:08:12.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:08:12.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:08:12.218 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:08:12.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:08:12.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:08:12.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:08:12.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:08:12.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:08:12.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:08:12.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:08:12.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:08:12.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:08:12.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:08:12.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:08:12.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T17:08:12.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:08:12.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:08:12.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:08:12.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T17:08:12.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:08:12.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:08:12.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-17T17:08:12.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T17:08:12.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:08:12.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:08:12.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.87940 2024-09-17T17:08:12.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T17:08:12.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T17:08:12.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T17:08:12.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-17T17:08:12.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-17T17:08:12.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-17T17:08:12.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T17:08:12.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T17:08:12.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T17:08:12.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-17T17:08:12.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-17T17:08:12.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T17:08:12.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:08:12.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:08:12.690 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:08:12.690 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:08:12.690 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:08:12.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:08:12.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T17:08:12.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T17:08:12.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T17:08:12.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:08:12.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T17:08:12.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T17:08:12.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T17:08:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T17:08:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T17:08:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T17:08:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:08:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:08:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:08:12.884 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T17:08:12.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:08:12.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get fsid 2024-09-17T17:08:12.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T17:08:12.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:08:12.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T17:08:12.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T17:08:12.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T17:08:12.964 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T17:08:12.965 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T17:08:12.965 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T17:08:12.965 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:08:12.965 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:08:12.965 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:08:12.965 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T17:08:12.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:08:12.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.87940/ceph-mon.a.asok config get mon_host 2024-09-17T17:08:13.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-17T17:08:13.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-17T17:08:13.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T17:08:13.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T17:08:13.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T17:08:13.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-17T17:08:13.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T17:08:13.896 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T17:08:13.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:08:13.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:08:13.897 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:08:13.897 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:08:13.897 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:08:13.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:08:13.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:08:13.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:08:13.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-17T17:08:13.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T17:08:14.206 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' already exists 2024-09-17T17:08:14.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T17:08:15.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-17T17:08:15.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T17:08:15.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T17:08:15.219 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:08:15.219 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:08:15.219 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:08:15.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-mon.a.asok 2024-09-17T17:08:15.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-17T17:08:15.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-mon.a.asok log flush 2024-09-17T17:08:15.290 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T17:08:15.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-17T17:08:15.299 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T17:08:12.721+0000 7f0838922c80 0 load: jerasure load: lrc 2024-09-17T17:08:15.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_1 td/test-erasure-eio 2024-09-17T17:08:15.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:364: TEST_rados_get_with_subreadall_eio_shard_1: local dir=td/test-erasure-eio 2024-09-17T17:08:15.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:365: TEST_rados_get_with_subreadall_eio_shard_1: local shard_id=1 2024-09-17T17:08:15.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_rados_get_with_subreadall_eio_shard_1: setup_osds 4 2024-09-17T17:08:15.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-17T17:08:15.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-17T17:08:15.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-17T17:08:15.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-17T17:08:15.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T17:08:15.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-17T17:08:15.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T17:08:15.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:08:15.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T17:08:15.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:08:15.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-17T17:08:15.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:08:15.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:08:15.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:08:15.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:08:15.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-17T17:08:15.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-17T17:08:15.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:08:15.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:08:15.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:08:15.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:08:15.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:08:15.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:08:15.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:08:15.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:08:15.307 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:08:15.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:08:15.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:08:15.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:08:15.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:08:15.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:08:15.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:08:15.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:08:15.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:08:15.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:08:15.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:08:15.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:08:15.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:08:15.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:08:15.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-17T17:08:15.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:08:15.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=595a546d-7f68-41dd-9d25-161733b9b58e 2024-09-17T17:08:15.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 595a546d-7f68-41dd-9d25-161733b9b58e' 2024-09-17T17:08:15.308 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 595a546d-7f68-41dd-9d25-161733b9b58e 2024-09-17T17:08:15.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:08:15.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB/t+lmDHpHExAAW8cX6bwey2NjUtLRlY4i8A== 2024-09-17T17:08:15.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB/t+lmDHpHExAAW8cX6bwey2NjUtLRlY4i8A=="}' 2024-09-17T17:08:15.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 595a546d-7f68-41dd-9d25-161733b9b58e -i td/test-erasure-eio/0/new.json 2024-09-17T17:08:15.582 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:08:15.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-17T17:08:15.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/t+lmDHpHExAAW8cX6bwey2NjUtLRlY4i8A== --osd-uuid 595a546d-7f68-41dd-9d25-161733b9b58e 2024-09-17T17:08:15.622 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:15.620+0000 7f9096fdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:08:15.651 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:15.649+0000 7f9096fdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:08:15.652 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:15.650+0000 7f9096fdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:08:15.653 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:15.651+0000 7f9096fdd640 -1 bdev(0x55c2f71c3c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-17T17:08:15.653 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:15.651+0000 7f9096fdd640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-17T17:08:21.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-17T17:08:21.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:08:21.602 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-09-17T17:08:21.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T17:08:21.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:08:22.136 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-09-17T17:08:22.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T17:08:22.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:08:22.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:08:22.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:08:22.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:08:22.163 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:22.160+0000 7f7f29e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:08:22.175 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:22.173+0000 7f7f29e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:08:22.177 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:22.174+0000 7f7f29e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:08:22.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T17:08:22.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:08:22.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T17:08:22.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:08:22.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:08:22.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:22.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:08:22.599 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:08:22.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:22.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:23.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:23.519 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:23.516+0000 7f7f29e14640 -1 Falling back to public interface 2024-09-17T17:08:24.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:24.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:24.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:08:24.025 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:08:24.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:24.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:24.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:25.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:25.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:25.459 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:08:25.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:08:25.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:25.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:25.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:25.978 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:25.975+0000 7f7f29e14640 -1 osd.0 0 log_to_monitors true 2024-09-17T17:08:26.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:26.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:26.859 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:08:26.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:08:26.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:26.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:27.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:28.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:28.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:28.341 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:08:28.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:08:28.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:28.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:28.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:29.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:29.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:29.847 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T17:08:29.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:08:29.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:29.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:30.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:31.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:31.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:31.248 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T17:08:31.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:08:31.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:31.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:31.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:32.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:32.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:32.648 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T17:08:32.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:08:32.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:32.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:33.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:34.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:34.181 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T17:08:34.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:34.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:08:34.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:34.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:34.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:35.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:35.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:35.581 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T17:08:35.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:08:35.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:35.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:35.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:36.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:36.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:36.980 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T17:08:36.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:08:36.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:36.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:37.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:38.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:38.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:38.568 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:08:38.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:08:38.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:38.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:39.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:40.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:40.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:40.303 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:08:40.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:08:40.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:40.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:41.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:42.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:42.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:42.426 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:08:42.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:08:42.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:42.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:43.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:44.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:44.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:44.270 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:08:44.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:08:44.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:44.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:45.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:46.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:46.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:46.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:08:46.104 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:08:46.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:46.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:46.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:47.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:47.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:47.825 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:08:47.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:08:47.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:47.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:48.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:49.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:49.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:49.653 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:08:49.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:08:49.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:49.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:08:50.100 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2591624922,v1:127.0.0.1:6803/2591624922] [v2:127.0.0.1:6804/2591624922,v1:127.0.0.1:6805/2591624922] exists,up 595a546d-7f68-41dd-9d25-161733b9b58e 2024-09-17T17:08:50.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:08:50.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:08:50.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:08:50.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T17:08:50.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-17T17:08:50.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T17:08:50.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:08:50.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T17:08:50.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:08:50.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-17T17:08:50.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:08:50.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:08:50.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:08:50.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:08:50.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-17T17:08:50.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-17T17:08:50.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:08:50.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:08:50.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:08:50.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:08:50.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:08:50.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:08:50.105 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:08:50.105 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:08:50.105 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:08:50.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:08:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:08:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:08:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:08:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:08:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:08:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:08:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:08:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:08:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:08:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:08:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:08:50.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:08:50.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-17T17:08:50.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:08:50.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d60390ea-7609-4257-b763-d37f75d221e0 2024-09-17T17:08:50.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d60390ea-7609-4257-b763-d37f75d221e0' 2024-09-17T17:08:50.107 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 d60390ea-7609-4257-b763-d37f75d221e0 2024-09-17T17:08:50.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:08:50.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCit+lm1F8nBxAAtmSdqs2/0QblaNxPXob6jA== 2024-09-17T17:08:50.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCit+lm1F8nBxAAtmSdqs2/0QblaNxPXob6jA=="}' 2024-09-17T17:08:50.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d60390ea-7609-4257-b763-d37f75d221e0 -i td/test-erasure-eio/1/new.json 2024-09-17T17:08:50.570 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:08:50.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-17T17:08:50.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCit+lm1F8nBxAAtmSdqs2/0QblaNxPXob6jA== --osd-uuid d60390ea-7609-4257-b763-d37f75d221e0 2024-09-17T17:08:50.611 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:50.609+0000 7f0732fc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:08:50.622 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:50.620+0000 7f0732fc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:08:50.623 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:50.621+0000 7f0732fc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:08:50.623 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:50.622+0000 7f0732fc5640 -1 bdev(0x56249e9bbc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-17T17:08:50.624 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:50.622+0000 7f0732fc5640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-17T17:08:56.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-17T17:08:56.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:08:56.309 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-09-17T17:08:56.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T17:08:56.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:08:56.844 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-09-17T17:08:56.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T17:08:56.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:08:56.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:08:56.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:08:56.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:08:56.871 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:56.869+0000 7f1196e16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:08:56.883 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:56.881+0000 7f1196e16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:08:56.884 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:56.883+0000 7f1196e16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:08:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T17:08:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:08:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T17:08:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:08:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:08:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:08:57.262 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:08:57.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:57.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:08:57.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:08:57.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:08:57.729+0000 7f1196e16640 -1 Falling back to public interface 2024-09-17T17:08:58.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:08:58.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:08:58.684 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:08:58.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:08:58.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:08:58.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:08:59.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:00.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:00.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:00.118 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:09:00.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:09:00.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:00.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:00.264 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:09:00.262+0000 7f1196e16640 -1 osd.1 0 log_to_monitors true 2024-09-17T17:09:00.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:01.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:01.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:01.581 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:09:01.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:09:01.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:01.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:02.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:03.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:03.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:03.178 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:09:03.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:09:03.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:03.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:03.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:04.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:04.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:04.607 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T17:09:04.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:09:04.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:04.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:05.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:06.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:06.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:06.093 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T17:09:06.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:09:06.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:06.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:06.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:07.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:07.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:07.520 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T17:09:07.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:09:07.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:07.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:07.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:08.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:08.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:08.921 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T17:09:08.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:09:08.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:08.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:09.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:10.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:10.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:10.404 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T17:09:10.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:09:10.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:10.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:10.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:11.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:11.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:11.824 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T17:09:11.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:09:11.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:11.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:12.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:13.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:13.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:13.263 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:09:13.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:09:13.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:13.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:14.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:15.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:15.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:15.110 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:09:15.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:09:15.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:15.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:16.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:17.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:17.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:17.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:09:17.081 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:09:17.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:17.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:17.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:18.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:18.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:18.835 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:09:18.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:09:18.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:18.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:19.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:20.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:20.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:20.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:09:20.561 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:09:20.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:20.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:20.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:21.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:21.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:21.961 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:09:21.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:09:21.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:21.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:22.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:23.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:23.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:23.943 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:09:23.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:09:23.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:23.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:09:24.505 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1870875154,v1:127.0.0.1:6811/1870875154] [v2:127.0.0.1:6812/1870875154,v1:127.0.0.1:6813/1870875154] exists,up d60390ea-7609-4257-b763-d37f75d221e0 2024-09-17T17:09:24.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:09:24.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:09:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:09:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T17:09:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-17T17:09:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T17:09:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:09:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T17:09:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:09:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T17:09:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:09:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:09:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:09:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:09:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T17:09:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T17:09:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:09:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:09:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:09:24.512 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:09:24.512 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:09:24.512 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:09:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:09:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:09:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:09:24.512 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:09:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:09:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:09:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:09:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:09:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:09:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:09:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:09:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:09:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:09:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:09:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:09:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:09:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T17:09:24.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:09:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=85fb74e4-6b8a-497a-80d6-19e5dde97144 2024-09-17T17:09:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 85fb74e4-6b8a-497a-80d6-19e5dde97144' 2024-09-17T17:09:24.514 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 85fb74e4-6b8a-497a-80d6-19e5dde97144 2024-09-17T17:09:24.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:09:24.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDEt+lmx3ZRHxAA51/u2PztDXI+KVAt10/C9A== 2024-09-17T17:09:24.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDEt+lmx3ZRHxAA51/u2PztDXI+KVAt10/C9A=="}' 2024-09-17T17:09:24.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 85fb74e4-6b8a-497a-80d6-19e5dde97144 -i td/test-erasure-eio/2/new.json 2024-09-17T17:09:25.116 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:09:25.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-17T17:09:25.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDEt+lmx3ZRHxAA51/u2PztDXI+KVAt10/C9A== --osd-uuid 85fb74e4-6b8a-497a-80d6-19e5dde97144 2024-09-17T17:09:25.158 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:09:25.156+0000 7ff5bb7c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:09:25.166 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:09:25.164+0000 7ff5bb7c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:09:25.167 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:09:25.165+0000 7ff5bb7c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:09:25.167 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:09:25.165+0000 7ff5bb7c5640 -1 bdev(0x558bfef55c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-17T17:09:25.167 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:09:25.165+0000 7ff5bb7c5640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-17T17:09:31.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-17T17:09:31.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:09:31.571 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-09-17T17:09:31.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T17:09:31.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:09:32.113 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T17:09:32.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T17:09:32.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:09:32.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:09:32.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:09:32.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:09:32.141 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:09:32.139+0000 7f753362c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:09:32.162 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:09:32.160+0000 7f753362c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:09:32.163 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:09:32.161+0000 7f753362c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:09:32.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T17:09:32.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:09:32.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T17:09:32.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:09:32.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:09:32.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:32.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:09:32.539 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:09:32.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:32.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:32.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:09:33.249+0000 7f753362c640 -1 Falling back to public interface 2024-09-17T17:09:33.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:33.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:33.962 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:09:33.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:09:33.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:33.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:34.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:35.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:35.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:35.381 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:09:35.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:09:35.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:35.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:35.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:36.076 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:09:36.074+0000 7f753362c640 -1 osd.2 0 log_to_monitors true 2024-09-17T17:09:36.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:36.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:36.780 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:09:36.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:09:36.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:36.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:37.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:38.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:38.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:38.318 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:09:38.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:09:38.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:38.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:38.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:39.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:39.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:39.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:09:39.848 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T17:09:39.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:39.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:40.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:41.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:41.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:41.498 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T17:09:41.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:09:41.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:41.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:41.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:42.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:42.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:42.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:09:42.899 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T17:09:42.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:42.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:43.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:44.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:44.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:44.370 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T17:09:44.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:09:44.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:44.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:44.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:45.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:45.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:45.833 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T17:09:45.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:09:45.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:45.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:46.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:47.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:47.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:47.289 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T17:09:47.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:09:47.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:47.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:47.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:48.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:48.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:48.695 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:09:48.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:09:48.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:48.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:49.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:50.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:50.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:50.455 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:09:50.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:09:50.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:50.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:51.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:52.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:52.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:52.106 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:09:52.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:09:52.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:52.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:52.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:53.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:53.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:53.960 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:09:53.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:09:53.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:53.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:54.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:55.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:55.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:55.885 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:09:55.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:09:55.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:55.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:56.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:57.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:57.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:57.687 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:09:57.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:09:57.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:57.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:09:58.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:09:59.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:09:59.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:09:59.349 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:09:59.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:09:59.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:09:59.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:10:00.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:01.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:01.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:01.196 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T17:10:01.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T17:10:01.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:01.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:10:01.728 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1876005775,v1:127.0.0.1:6819/1876005775] [v2:127.0.0.1:6820/1876005775,v1:127.0.0.1:6821/1876005775] exists,up 85fb74e4-6b8a-497a-80d6-19e5dde97144 2024-09-17T17:10:01.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:10:01.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:10:01.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:10:01.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-17T17:10:01.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-17T17:10:01.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-17T17:10:01.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:10:01.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T17:10:01.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:10:01.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-17T17:10:01.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:10:01.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:10:01.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:10:01.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:10:01.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-17T17:10:01.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-17T17:10:01.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:10:01.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:10:01.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:10:01.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:10:01.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:10:01.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:10:01.733 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:10:01.733 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:10:01.734 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:10:01.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:10:01.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:10:01.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:10:01.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:10:01.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:10:01.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:10:01.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:10:01.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:10:01.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:10:01.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:10:01.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:10:01.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:10:01.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:10:01.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-17T17:10:01.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:10:01.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cb3dccc8-f2bc-4749-8376-5bdc03a7bf1d 2024-09-17T17:10:01.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 cb3dccc8-f2bc-4749-8376-5bdc03a7bf1d' 2024-09-17T17:10:01.735 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 cb3dccc8-f2bc-4749-8376-5bdc03a7bf1d 2024-09-17T17:10:01.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:10:01.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDpt+lmseClLBAAaZ3IS3nr1o8Mbm6R15wTFQ== 2024-09-17T17:10:01.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDpt+lmseClLBAAaZ3IS3nr1o8Mbm6R15wTFQ=="}' 2024-09-17T17:10:01.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cb3dccc8-f2bc-4749-8376-5bdc03a7bf1d -i td/test-erasure-eio/3/new.json 2024-09-17T17:10:02.298 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:10:02.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-17T17:10:02.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDpt+lmseClLBAAaZ3IS3nr1o8Mbm6R15wTFQ== --osd-uuid cb3dccc8-f2bc-4749-8376-5bdc03a7bf1d 2024-09-17T17:10:02.340 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:10:02.337+0000 7fd1f96dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:10:02.348 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:10:02.345+0000 7fd1f96dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:10:02.349 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:10:02.346+0000 7fd1f96dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:10:02.350 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:10:02.347+0000 7fd1f96dc640 -1 bdev(0x556217eb4800 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-17T17:10:02.350 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:10:02.347+0000 7fd1f96dc640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-17T17:10:08.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-17T17:10:08.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:10:08.024 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-09-17T17:10:08.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T17:10:08.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:10:08.565 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-09-17T17:10:08.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T17:10:08.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:10:08.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:10:08.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:10:08.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:10:08.592 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:10:08.589+0000 7f45081fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:10:08.612 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:10:08.610+0000 7f45081fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:10:08.613 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:10:08.611+0000 7f45081fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:10:09.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T17:10:09.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:10:09.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T17:10:09.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:10:09.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:10:09.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:09.026 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:10:09.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:10:09.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:09.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:09.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:09.960 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:10:09.958+0000 7f45081fd640 -1 Falling back to public interface 2024-09-17T17:10:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:10:10.416 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:10:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:10.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:11.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:11.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:11.847 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:10:11.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:10:11.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:11.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:12.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:13.134 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:10:13.132+0000 7f45081fd640 -1 osd.3 0 log_to_monitors true 2024-09-17T17:10:13.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:13.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:13.291 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:10:13.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:10:13.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:13.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:13.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:14.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:14.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:14.840 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-09-17T17:10:14.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:10:14.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:14.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:15.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:16.388 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-09-17T17:10:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:10:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:16.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:17.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:17.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:17.845 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-09-17T17:10:17.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:10:17.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:17.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:18.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:19.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:19.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:19.277 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-09-17T17:10:19.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:10:19.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:19.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:19.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:20.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:20.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:20.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:10:20.683 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-09-17T17:10:20.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:20.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:21.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:22.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:22.215 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-09-17T17:10:22.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:22.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:10:22.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:22.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:22.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:23.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:23.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:23.632 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-09-17T17:10:23.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:10:23.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:23.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:24.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:25.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:25.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:25.145 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-09-17T17:10:25.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:10:25.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:25.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:25.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:26.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:26.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:26.569 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-09-17T17:10:26.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:10:26.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:26.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:27.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:28.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:28.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:28.100 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-09-17T17:10:28.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:10:28.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:28.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:29.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:30.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:30.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:30.278 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-09-17T17:10:30.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:10:30.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:30.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:31.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:32.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:32.239 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-09-17T17:10:32.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:32.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:10:32.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:32.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:33.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:33.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:33.975 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-09-17T17:10:33.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:10:33.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:33.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:34.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:35.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:35.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:35.776 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-09-17T17:10:35.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:10:35.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:35.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:37.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:38.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:38.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:38.044 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-09-17T17:10:38.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T17:10:38.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:38.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:10:38.539 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/51179727,v1:127.0.0.1:6827/51179727] [v2:127.0.0.1:6828/51179727,v1:127.0.0.1:6829/51179727] exists,up cb3dccc8-f2bc-4749-8376-5bdc03a7bf1d 2024-09-17T17:10:38.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:10:38.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:10:38.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:10:38.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-17T17:10:38.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T17:10:38.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T17:10:38.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:10:38.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:10:38.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:10:38.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.0.asok 2024-09-17T17:10:38.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-17T17:10:38.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.0.asok log flush 2024-09-17T17:10:38.620 INFO:tasks.workunit.client.0.smithi162.stdout:{} 2024-09-17T17:10:38.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-17T17:10:38.630 INFO:tasks.workunit.client.0.smithi162.stdout:2024-09-17T17:08:23.520+0000 7f7f29e14640 0 load: jerasure load: lrc 2024-09-17T17:10:38.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:369: TEST_rados_get_with_subreadall_eio_shard_1: local poolname=pool-jerasure 2024-09-17T17:10:38.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:370: TEST_rados_get_with_subreadall_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-17T17:10:38.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T17:10:38.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-17T17:10:38.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-17T17:10:38.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-17T17:10:38.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-17T17:10:38.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-17T17:10:38.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-17T17:10:39.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-17T17:10:39.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-17T17:10:40.707 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' already exists 2024-09-17T17:10:40.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T17:10:41.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-17T17:10:41.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T17:10:41.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T17:10:41.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T17:10:41.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T17:10:41.721 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T17:10:41.721 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T17:10:41.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T17:10:41.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T17:10:41.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T17:10:41.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T17:10:41.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T17:10:41.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T17:10:41.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T17:10:41.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T17:10:41.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T17:10:42.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T17:10:42.247 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:10:42.247 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T17:10:42.247 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T17:10:42.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T17:10:42.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:10:42.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T17:10:42.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803805 2024-09-17T17:10:42.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803805 2024-09-17T17:10:42.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805' 2024-09-17T17:10:42.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:10:42.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T17:10:42.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574873 2024-09-17T17:10:42.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574873 2024-09-17T17:10:42.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805 1-55834574873' 2024-09-17T17:10:42.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:10:42.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T17:10:42.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378639 2024-09-17T17:10:42.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378639 2024-09-17T17:10:42.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805 1-55834574873 2-81604378639' 2024-09-17T17:10:42.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:10:42.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T17:10:42.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084298 2024-09-17T17:10:42.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084298 2024-09-17T17:10:42.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805 1-55834574873 2-81604378639 3-120259084298' 2024-09-17T17:10:42.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:10:42.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803805 2024-09-17T17:10:42.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:10:42.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T17:10:42.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803805 2024-09-17T17:10:42.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:10:42.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803805 2024-09-17T17:10:42.806 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803805 2024-09-17T17:10:42.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803805' 2024-09-17T17:10:42.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:10:43.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803803 -lt 25769803805 2024-09-17T17:10:43.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T17:10:44.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T17:10:44.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:10:44.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803805 -lt 25769803805 2024-09-17T17:10:44.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:10:44.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574873 2024-09-17T17:10:44.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:10:44.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T17:10:44.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574873 2024-09-17T17:10:44.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:10:44.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574873 2024-09-17T17:10:44.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574873' 2024-09-17T17:10:44.662 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574873 2024-09-17T17:10:44.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T17:10:45.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574873 -lt 55834574873 2024-09-17T17:10:45.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:10:45.094 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378639 2024-09-17T17:10:45.094 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:10:45.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T17:10:45.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378639 2024-09-17T17:10:45.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:10:45.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378639 2024-09-17T17:10:45.097 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 81604378639 2024-09-17T17:10:45.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378639' 2024-09-17T17:10:45.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T17:10:45.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378639 -lt 81604378639 2024-09-17T17:10:45.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:10:45.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084298 2024-09-17T17:10:45.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:10:45.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T17:10:45.486 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084298 2024-09-17T17:10:45.486 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:10:45.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084298 2024-09-17T17:10:45.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084298' 2024-09-17T17:10:45.487 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084298 2024-09-17T17:10:45.487 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T17:10:45.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084299 -lt 120259084298 2024-09-17T17:10:45.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T17:10:45.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:10:45.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:10:46.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T17:10:46.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T17:10:46.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T17:10:46.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T17:10:46.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T17:10:46.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T17:10:46.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T17:10:46.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T17:10:46.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T17:10:46.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T17:10:46.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:10:46.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:10:47.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T17:10:47.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T17:10:47.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T17:10:47.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:372: TEST_rados_get_with_subreadall_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 recovery 2024-09-17T17:10:47.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-17T17:10:47.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-17T17:10:47.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-17T17:10:47.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-17T17:10:47.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-17T17:10:47.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-17T17:10:47.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2024-09-17T17:10:47.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-17T17:10:47.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-87940-1 2024-09-17T17:10:47.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-87940-1 2024-09-17T17:10:47.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-17T17:10:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-17T17:10:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-87940-1 2024-09-17T17:10:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:10:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-17T17:10:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:10:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-17T17:10:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:10:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-17T17:10:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-17T17:10:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-17T17:10:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-87940-1 td/test-erasure-eio/ORIGINAL 2024-09-17T17:10:47.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-87940-1 td/test-erasure-eio 1 2024-09-17T17:10:47.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-17T17:10:47.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T17:10:47.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T17:10:47.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T17:10:47.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-17T17:10:47.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T17:10:47.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-87940-1 2024-09-17T17:10:47.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T17:10:47.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-17T17:10:47.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T17:10:47.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-17T17:10:47.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T17:10:47.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-87940-1 2024-09-17T17:10:47.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T17:10:47.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-87940-1 2024-09-17T17:10:47.559 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-87940-1 2024-09-17T17:10:47.559 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T17:10:48.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T17:10:48.012 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:10:48.012 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T17:10:48.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T17:10:48.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-17T17:10:48.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T17:10:48.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-17T17:10:48.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-17T17:10:48.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/1/type 2024-09-17T17:10:48.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T17:10:48.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-17T17:10:48.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T17:10:48.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-17T17:10:48.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T17:10:48.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T17:10:48.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T17:10:48.015 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-17T17:10:48.015 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T17:10:48.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T17:10:48.016 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:10:48.016 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:10:48.016 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:10:48.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.1.asok 2024-09-17T17:10:48.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87940/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-17T17:10:48.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T17:10:48.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T17:10:48.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T17:10:48.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-17T17:10:48.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T17:10:48.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T17:10:48.101 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:10:48.101 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:10:48.101 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:10:48.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.87940/ceph-osd.1.asok 2024-09-17T17:10:48.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T17:10:48.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87940/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-87940-1 1 2024-09-17T17:10:48.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-87940-1 2024-09-17T17:10:48.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-17T17:10:48.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-17T17:10:48.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-87940-1 2024-09-17T17:10:48.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-17T17:10:48.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-17T17:10:48.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-87940-1 td/test-erasure-eio/COPY 2024-09-17T17:10:48.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-17T17:10:48.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-17T17:10:48.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2024-09-17T17:10:48.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-87940-1 2024-09-17T17:10:48.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T17:10:48.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-87940-1 2024-09-17T17:10:48.238 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-87940-1 2024-09-17T17:10:48.238 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T17:10:48.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T17:10:48.655 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:10:48.655 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-09-17T17:10:48.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-17T17:10:48.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2024-09-17T17:10:48.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2024-09-17T17:10:48.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2024-09-17T17:10:48.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-17T17:10:48.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:10:48.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:10:48.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:10:48.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:10:48.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:10:48.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:10:48.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2024-09-17T17:10:50.003 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already out. 2024-09-17T17:10:50.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-87940-1 2024-09-17T17:10:50.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2024-09-17T17:10:50.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-17T17:10:50.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-87940-1 2024-09-17T17:10:50.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-87940-1 2024-09-17T17:10:50.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T17:10:50.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T17:10:50.710 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:10:50.710 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-09-17T17:10:50.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 0 2024-09-17T17:10:50.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2024-09-17T17:10:51.791 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 is already in. 2024-09-17T17:10:51.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2024-09-17T17:10:51.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-17T17:10:51.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T17:10:51.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T17:10:51.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T17:10:51.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-17T17:10:51.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-17T17:10:51.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:10:51.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:10:51.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:10:51.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-17T17:10:51.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-17T17:10:51.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T17:10:51.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T17:10:51.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-17T17:10:51.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T17:10:51.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:10:51.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:10:51.807 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:10:51.807 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:10:51.807 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:10:51.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:10:51.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' 2024-09-17T17:10:51.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:10:51.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-17T17:10:51.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-17T17:10:51.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:10:51.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:10:51.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:10:51.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:10:51.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T17:10:51.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T17:10:51.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-17T17:10:51.808 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-09-17T17:10:51.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T17:10:51.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=87742e7f-af32-486d-8265-12e0a688c8f5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87940/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:10:51.809 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-17T17:10:51.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T17:10:51.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T17:10:51.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T17:10:51.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T17:10:51.833 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:10:51.830+0000 7f5b00c0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:10:51.835 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:10:51.832+0000 7f5b00c0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:10:51.836 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:10:51.833+0000 7f5b00c0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:10:52.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T17:10:52.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:10:52.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T17:10:52.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:10:52.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:10:52.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:52.439 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-09-17T17:10:52.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:10:52.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:52.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:10:52.685 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:10:52.682+0000 7f5b00c0c640 -1 Falling back to public interface 2024-09-17T17:10:53.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:54.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:54.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:54.077 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-09-17T17:10:54.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:10:54.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:54.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:10:54.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:55.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:55.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:55.537 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-09-17T17:10:55.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:10:55.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:55.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:10:55.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:10:55.996 INFO:tasks.workunit.client.0.smithi162.stderr:2024-09-17T17:10:55.991+0000 7f5b00c0c640 -1 osd.2 35 log_to_monitors true 2024-09-17T17:10:56.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:10:56.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:10:56.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:10:56.942 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-09-17T17:10:56.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:10:56.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:10:57.397 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 48 up_thru 19 down_at 36 last_clean_interval [19,35) [v2:127.0.0.1:6818/2177926940,v1:127.0.0.1:6819/2177926940] [v2:127.0.0.1:6820/2177926940,v1:127.0.0.1:6821/2177926940] exists,up 85fb74e4-6b8a-497a-80d6-19e5dde97144 2024-09-17T17:10:57.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:10:57.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:10:57.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:10:57.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2024-09-17T17:10:57.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T17:10:57.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T17:10:57.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T17:10:57.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T17:10:57.400 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T17:10:57.400 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T17:10:57.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T17:10:57.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T17:10:57.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T17:10:57.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T17:10:57.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T17:10:57.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T17:10:57.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T17:10:57.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T17:10:57.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T17:10:57.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T17:10:57.882 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-09-17T17:10:57.882 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-09-17T17:10:57.882 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-09-17T17:10:57.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T17:10:57.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:10:57.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T17:10:57.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803810 2024-09-17T17:10:57.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803810 2024-09-17T17:10:57.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803810' 2024-09-17T17:10:57.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:10:57.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T17:10:58.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574878 2024-09-17T17:10:58.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574878 2024-09-17T17:10:58.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803810 1-55834574878' 2024-09-17T17:10:58.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:10:58.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T17:10:58.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=206158430211 2024-09-17T17:10:58.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 206158430211 2024-09-17T17:10:58.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803810 1-55834574878 2-206158430211' 2024-09-17T17:10:58.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T17:10:58.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T17:10:58.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084303 2024-09-17T17:10:58.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084303 2024-09-17T17:10:58.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803810 1-55834574878 2-206158430211 3-120259084303' 2024-09-17T17:10:58.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:10:58.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803810 2024-09-17T17:10:58.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:10:58.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T17:10:58.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803810 2024-09-17T17:10:58.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:10:58.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803810 2024-09-17T17:10:58.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803810' 2024-09-17T17:10:58.405 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 25769803810 2024-09-17T17:10:58.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:10:58.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803808 -lt 25769803810 2024-09-17T17:10:58.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T17:10:59.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T17:10:59.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T17:11:00.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803811 -lt 25769803810 2024-09-17T17:11:00.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:11:00.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574878 2024-09-17T17:11:00.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:11:00.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T17:11:00.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574878 2024-09-17T17:11:00.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:11:00.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574878 2024-09-17T17:11:00.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574878' 2024-09-17T17:11:00.265 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 55834574878 2024-09-17T17:11:00.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T17:11:00.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574879 -lt 55834574878 2024-09-17T17:11:00.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:11:00.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-206158430211 2024-09-17T17:11:00.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:11:00.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T17:11:00.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-206158430211 2024-09-17T17:11:00.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:11:00.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=206158430211 2024-09-17T17:11:00.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 206158430211' 2024-09-17T17:11:00.660 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 206158430211 2024-09-17T17:11:00.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T17:11:01.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 206158430211 -lt 206158430211 2024-09-17T17:11:01.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T17:11:01.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084303 2024-09-17T17:11:01.083 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T17:11:01.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T17:11:01.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084303 2024-09-17T17:11:01.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T17:11:01.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084303 2024-09-17T17:11:01.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084303' 2024-09-17T17:11:01.085 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 120259084303 2024-09-17T17:11:01.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T17:11:01.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084304 -lt 120259084303 2024-09-17T17:11:01.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T17:11:01.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:11:01.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:11:02.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T17:11:02.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T17:11:02.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T17:11:02.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T17:11:02.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T17:11:02.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T17:11:02.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T17:11:02.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T17:11:02.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T17:11:02.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T17:11:02.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:11:02.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:11:02.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T17:11:02.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T17:11:02.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T17:11:02.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T17:11:02.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T17:11:02.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T17:11:03.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T17:11:03.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T17:11:03.059 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T17:11:03.059 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T17:11:03.059 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T17:11:03.059 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T17:11:03.059 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T17:11:03.059 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T17:11:03.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T17:11:03.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T17:11:03.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:11:03.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:11:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T17:11:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-17T17:11:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T17:11:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T17:11:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T17:11:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T17:11:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T17:11:03.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T17:11:03.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T17:11:04.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T17:11:04.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T17:11:04.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T17:11:04.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T17:11:04.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T17:11:04.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T17:11:04.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T17:11:04.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T17:11:04.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T17:11:04.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T17:11:04.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T17:11:04.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T17:11:04.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T17:11:05.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T17:11:05.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T17:11:05.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T17:11:05.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T17:11:05.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T17:11:05.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T17:11:05.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T17:11:05.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:374: TEST_rados_get_with_subreadall_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-17T17:11:05.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-17T17:11:05.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-17T17:11:06.220 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'pool-jerasure' does not exist 2024-09-17T17:11:06.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-17T17:11:06.727 INFO:tasks.workunit.client.0.smithi162.stderr:erasure-code-profile myprofile does not exist 2024-09-17T17:11:06.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-17T17:11:06.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T17:11:06.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:11:06.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T17:11:06.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:11:06.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:11:06.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:11:06.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:11:06.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:11:06.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:11:06.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:11:06.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:11:06.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:11:06.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:11:06.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:11:06.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:11:06.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:11:06.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:11:06.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:11:06.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:11:06.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:11:06.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:11:06.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T17:11:06.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:11:06.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:11:06.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:11:06.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T17:11:06.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:11:06.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:11:06.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-17T17:11:06.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/test-erasure-eio 0 2024-09-17T17:11:06.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-17T17:11:06.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-17T17:11:06.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-17T17:11:06.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:11:06.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:11:06.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:11:06.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:11:06.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:11:06.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:11:06.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:11:06.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:11:06.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:11:06.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:11:06.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:11:06.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:11:06.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:11:06.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:11:06.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:11:06.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:11:06.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:11:06.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-17T17:11:06.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-17T17:11:06.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:11:06.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:11:06.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.87940 2024-09-17T17:11:06.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.87940 2024-09-17T17:11:06.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:11:06.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:11:06.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-17T17:11:06.888 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T17:11:06.888 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T17:11:06.913 INFO:tasks.workunit:Stopping ['erasure-code'] on client.0... 2024-09-17T17:11:06.913 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-17T17:11:07.241 DEBUG:teuthology.parallel:result is None 2024-09-17T17:11:07.241 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T17:11:07.266 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T17:11:07.266 DEBUG:teuthology.orchestra.run.smithi162:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-17T17:11:07.333 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T17:11:07.333 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T17:11:07.349 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T17:11:07.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T17:11:07.423 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T17:11:07.439 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T17:11:07.439 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T17:11:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-09-17T17:11:07.459 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T17:11:07.459 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-09-17T17:11:07.459 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv01.front.sepia.ceph.com 3 6 377 32 +295us[ +284us] +/- 47ms 2024-09-17T17:11:07.459 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 32 +170us[ +160us] +/- 61ms 2024-09-17T17:11:07.459 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 167 -694us[ -712us] +/- 67ms 2024-09-17T17:11:07.459 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T17:11:07.459 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T17:11:07.468 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T17:11:07.470 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T17:11:07.487 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T17:11:07.558 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T17:11:07.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T17:11:07.586 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T17:11:07.597 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T17:11:07.617 INFO:teuthology.task.internal:Duration was 5051.983453 seconds 2024-09-17T17:11:07.617 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T17:11:07.629 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T17:11:07.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T17:11:07.668 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T17:11:07.873 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T17:11:07.873 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-09-17T17:11:07.874 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T17:11:07.940 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T17:11:07.940 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T17:11:08.007 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T17:11:08.007 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T17:11:08.146 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T17:11:08.155 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T17:11:08.155 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T17:11:08.180 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T17:11:08.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:11:08.244 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-09-17T17:11:08.257 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:11:08.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T17:11:08.313 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T17:11:08.322 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T17:11:08.324 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909006/remote/smithi162 2024-09-17T17:11:08.325 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T17:11:08.405 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T17:11:08.406 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T17:11:08.454 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T17:11:08.473 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T17:11:08.474 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T17:11:08.578 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T17:11:08.579 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T17:11:08.594 INFO:teuthology.orchestra.run.smithi162.stdout: 265336 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 17:11 /home/ubuntu/cephtest 2024-09-17T17:11:08.595 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T17:11:08.633 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T17:11:08.886 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} duration: 5051.983452796936 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-17T17:11:08.886 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T17:11:08.979 INFO:teuthology.run:pass